Enhancing Large Vision Language Models with Self-Training on Image Comprehension
Authors: Yihe Deng
Pan Lu
Fan Yin
Ziniu Hu
Sheng Shen
Quanquan Gu
James Zou
Kai-Wei Chang
Wei Wang
Venue: NeurIPS 2024 poster (2024)
Keywords: Large Vision-Language Models
multimodal preference alignment
self-training
synthetic data
Primary area: natural_language_processing

Abstract: Large vision language models (LVLMs) integrate large language models (LLMs) with pre-trained vision encoders, thereby activating the perception capability of the model to understand image inputs for different queries and conduct subsequent reasoning. Improving this capability requires high-quality vision-language data, which is costly and labor-intensive to acquire. Self-training approaches have been effective in single-modal settings to alleviate the need for labeled data by leveraging model's own generation. However, effective self-training remains a challenge regarding the unique visual perception and reasoning capability of LVLMs. To address this, we introduce **S**elf-**T**raining on **I**mage **C**omprehension (**STIC**), which emphasizes a self-training approach specifically for image comprehension. First, the model self-constructs a preference dataset for image descriptions using unlabeled images. Preferred responses are generated through a step-by-step prompt, while dis-preferred responses are generated from either corrupted images or misleading prompts. To further self-improve reasoning on the extracted visual information, we let the model reuse a small portion of existing instruction-tuning data and append its self-generated image descriptions to the prompts. We validate the effectiveness of STIC across seven different benchmarks, demonstrating substantial performance gains of 4.0% on average while using 70% less supervised fine-tuning data than the current method. Further studies dive into various components of STIC and highlight its potential to leverage vast quantities of unlabeled images for self-training.

Enhancing Large Vision Language Models with on Image Comprehension
Yihe Deng"!, Pan Lu Fan Yin', Ziniu Hu', Sheng Shen? Quanquan Gu', James Zou?, Kai- Chang', Wei Wang'
University California, Los Angeles
University of California, Berkeley Stanforr University
Abstract
Large vision language models (LVLMs) integrate large language models (LLMs) with pre-trained vision encoders, thereby activating the model's perception capa- bility to understand image inputs and conduct subsequent reasoning for different queries. Improving this capability requires high-quality vision-language data, which is costly and labor-intensive to acquire. Self-training approaches have been effective in single-modal settings to alleviate the need for labeled data by leveraging model's own generation. However, effective self-training remains chal- lenge regarding the unique visual perception and reasoning capability of LVLMs. To address this, introduce Self-Traie Self-Training Image Comprehension (STIC), which emphasizes a self-training approach specifically for image comprehension. First, the model self-constructs preference dataset for image descriptions us- ing unlabeled images. Preferred responses are generated through a step-by-step prompt, while dis-preferred responses generated from either corrupted images or misleading prompts. To further self- self-improve reasoning on the extracted visual in- formation, we let the model reuse small portion of existing instruction-tuning data and append its self-generated image descriptions to the prompts. We validate the effectiveness of STIC across seven different benchmarks, demonstrating substantial performance gains of 4.0% on average while using 70% less supervised fine-tumin fine-tuning data than the current method. Further studies investigate various components of STIC and highlight its potential to leverage vast quantities of unlabeled images for self-training. Code and data are made publicly available on GitHHu.
Introduction
In recent years, we have witnessed remarkable advancements in large language models (LLMs), such as GPT-4 (OpenAI, 2023a) and the LLaMA family (Touvron et al., 2023a.b). The increasing importance of processing multimodal inputs, including images and text, has significantly driven progress in vision language models (Radford et al., 2021; Jia et al., 2021b; Goel et al., 2022). Leveraging the powerful language understanding and generation capabilities of LLMs, researchers have advanced vision language models into large vision language models (LVLMss. This enhancement is achieved by integrating LLMs with image encoders (Radford et al., 2021; Li et al., 2023a). which were pre-trained on large-scale image-text pairs to ensure alignment between the two domains. For instance, LLaVA (Liu al., 2023b) integrates vision encoder from CLIP (Radford et al., 2021) with the LLM Vicuna (Chiang et al., 2023b), which is further fine-tuned on carefully constructed vision-language instructional datasets activate the model's perception capability of capturing the vision information according to different queries. This recent development has substantially expanded the requirement for large-scale instruction fine-tuning data for LVLMs (Gao et al., 2023b; Bai et al., 2023; Chen et al., 2023b; Gao et al., 2024; Anthropic, 2024; McKinzie et al., 2024).
Equual contribution. Contribution statement is provided after Scction 7.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Accuracy Improvement (%)
+6.4
STIC (LLaVA-V1.6)
Gasoline
Self Serve
Query: How many gallons of supreme gasoline can I get with $50?
+5.1+4,9 4.9
3.49 9
5
Reqular
Base (LLaVA-v1.6 7B);
4-
+4.1 avg.
+4.0
Based on the current gas prices displayed on the sign, you can get approximately 3.65 gallons of supreme gasoline with $50.
3.55
3-
+2.8
+2.1
Plus
+1.9
3.65
Supreme
STIC (LLaVA-v1.6 7B):
With $50, you can get approximately 13.69 gallons of supreme gasoline, as indicated by the price of $3.65 per gallon on the sign.
ScienceGA
ChanoA
TextvOA
MMBench
MM-vst.
Mathvista
LLa LLaVA-Bench Bench
Diesel 3.60 9
No.2
Figure I: Left: Accuracy improvement of our method, STIC, compared to the original LLaVA-v1.6 (Liu et al., 2024) seven benchmarks. Right: Response examples from the original LLaVA-v1.0 1.6 and STIC (LLaVA-v) 16), which enhances image comprehension and subsequent reasoning capabilities.
While LVLMs have shown promising results, a key challenge lies in the acquisition of high- fine-tuning data. Obtaining human-curated content at scale is often prohibitively expensive, especially for multi-modal data. Many recent studies resort to GPT-4V (OpenAI, 2023b) for generating or labeling high-quality vision-language fine-tuning data. However, this approach does not significantly reduce the cost (Liu et al., 2023b; Wu et al., 2024). For instance, using GPT-4V to generate 6k image descriptions with 1 tokens per output would cost approximately $200. There remains a pressing need for cost-effective methods to gather tne-ttuing data to further enhance LVLMs
To tackle the data acquisition bottleneck in multi-modality, we propose Self-Training Image Comprehension (STIC). Our method is inspired by the recent success of self-training (Chen et al., 2024; Yuan et al., 2024; Fränken et al., 2024; Rosset et al., 2024) in LLMs, which leverages self- generated data to improve their downstream performance. However, different from the text-only domain, the unique vision modality of LVLMs introduces new challenges, as LVLMs must understand the input image content before reasoning and responding to any related textual queries about the image. Therefore, the proposed STIC approach is novel two-stage self-training method that targets both image perception and reasoning over images and texts.
The overall framework is summarized in Figure 2. STIC specifically emphasizes the image compre- hension self-training of LVLMs where the model generates its own preference dataset focused on image description. The self-generan
dispreferred response is obtained by gathering model responses from either (1) prompts likely to elicit inaccurate responses or (2) corrupted images. The preferred responses are collected via detailed prompt that guides the model through a step-by-step image description process. Figure 3 shows examples of such generated responses. During fine-tuning, we consider DPO loss (Rafailov et al., 2023) with an additional regularized term explicitly emphasizing the preferred response. Lastly, we allow the model to self-improve its reasoning ability based on its own extracted image information by reusing small amount of existing instruction data and appending its self- ceff-generated image descriptions to the prompts. We refer to this second stage as deseription-infuseed fine-tuning. Notably, our STIC approach does not require pre-labeled information of the images, which contrasts to the recent works that rely on such information for constructing vision-language preference data (Zhou et al., 2024).
To demonstrate the effectiveness of STIC, we conduct extensive experiments on seven vision-languaae benchmarks, including ScienceQA (Lu et al., 2022), TexiVQA (Singh et al., 2019), ChartQA (Masry et al., 2022), LLa (Liu et al., 2023a), MMBench (Liu et al., 2023c), MM-Vet (Yu et al., 2023), and Math Vista (Lu et al., 2024). These benchmarks encompass scientific reasoning, math reasoning, optical character recognition (OCR), and conversation capabilities based on vision inputs, spanning various image sources such natural, chart, and text-rich images. We employ LLaVA- v1.6 (Liu et al., 2024) as the primary base LVLM for our experiments and unitize 6k images from MSCOCO (Lin et al., 2014) to construct the image description preference data. As depicted in Figure 1, STIC achieves consistent and significant performance improvements across these benchmarks, with an average accuracy gain of 4.0% over the base LVLM and notable gain of 6.4% on ScienceQA. We also provide an example of the different responses from the original LVLM and STIC in Figure 1, where STIC successfully identifies the key visual information and accurately reason with it. These results demonstrate the remarkable effectiveness of our image comprehension self-training approach in enhancing the visual perception capabilities of LVLMs

---

Good Prompting (step-by-step)
Stage 1: Image Comprehension Self-Training
Unlabeled Images
Pleuse provide a detailed description of the image, focusing the follow- ing aspects:
Preferred
Dis-preferred
subjects (people, animals, objects) in the imuge and describe what they ure doing.... ....
(1) Identify
Stage-1 LVLM
DPO alignment fine-tuming wi regulatization
Image Corruption
Bad Prompting
<Image descriptions dnstruction dResponse>
(1) Suggest and detail practical items or people that could logically inhabit the image's setting.
(2) Augment the sccac with details of
Used instruction fine-tuning data
potential events or Items. (3) Incorporatic elements that, though absent, would scamlessly fit into the
Supervised finc-tuning
15 Stage-2 LVLM
Stage 2: Deseription-Infused Fine-Tuning
context of the picture.
Figure 2: Framework overview of STIC, two-stage self-training algorithm focusing on the image comprehension capability of the LVLMs. In Stage 1, the base LVLM self-construct its preference dataset for image description using well-designed prompts. poorly dooly-des prompts, and distorted images. In Stage 2. a small portion of the previously used SFT data is recycled and infused with model-generated image descriptions to further fine-tune the base LVL.M
In addition, we explore the benefits of the various components of STIC. First, based on the description- infused fine-tuning stage that enhances the model's 's reasoning ability with self reffeeeertteed description, we show that further letting the model describe the image before responding to a query provides further improved reasoning capability. This results in notable improvement of 2.8% on ScienceQA and 1.1% on average as compared to direct responses to queries (Table 2). Moreover, we examine the impact of self- self-generated dispreferred responses, from either bad prompting or image corruption. By excluding these dispreferred responses and conducting SFT solely with preferred responses, we observed performance decrease of 2.5% on average across three benchmarks compared to STIC with the preference data (Table 3). This highlights the importance of the negative samples in the self-constructed preference data by STIC. We also assess the scalability of our self- self-training scheme. By increasing the amount of generated preference data from 6k to 12k, we show an even further improvement of STIC from 1.9% to 3.1% on LLa VA-Bench (Figure 6). This result suggests that STIC holds considerable potential for leveraging vast quantities of unlabeled images for self-training, given the immense availability of unlabeled image data. Lastly, our t-SNE visualization analysis shows that the closer the distribution between MSCOCO images, which we use for preference data construction, to images in downstream tasks, the more likely STIC results in higher performance gains (Figure 7).
The main contributions of this work are summarized as follows:
We propose STIC, novel two-stage self-training approach for LVLMs that focuses on enhancing their image comprehension capabilities by generating preference dataset for image description without relying on pre-labeled image information.
Through extensive experiments on seven diverse benchmarks, STIC demonstrates significant
performance gains over the base LVLM, achieving an average accuracy gain of 4.0%.
We explore the benefits of various components of STIC, highlighting its potential to leverage vast quantities of unlabeled images for self-training.
2 Related Work
Vision language models (VLMs). VLMS (Tan and Bansal, 2019; Li et al., 2019, 2020; Kim et al., 2021; Wang et al., 2022b; Bao al., 2022; Wang al., 2022a; Alayrac et al., 2022; Li et al., 2023b; Chen et al., 2022; Jia et al., 2021a; Shen et al., 2022; Singh et al., 2021), processing both images and text, are pivotal in a wide range of multimodal understanding and reasoning tasks, capable of generating text or encoding multimodal representations. These models have shown increasing proficiency in visual perception and textual reasoning, and are also capable of following complex instructions (OpenAI, 2023b; Team et al., 2023). Recent advancements in the field have been propelled by the availability of open-source large language models (LLMs) (Touvron et al., b:23a.b; Jiang et al., 2023) and innovative image encoders (Radford et al., 2021; Li et al., 2022). For instance, ILaVA (Liu et al., 2023b) combines vision encoder from CLIP (Radford et al., 2021) with the Vicuna LLM (Chiang et al., 2023b), and has been further fine-tuned vision-language instruction-
3

---

Task: Explain what is depicted in the photograph.
Preferred data (Good prompting: Provide a detailed description of the image, focusing on ...)
The image depicts a dynamic winter sports scene. The main subjects are a person engaged skiing, captured mid-air while performing a jump or trick, The skier is wearing a blue yellow jacket, black helmet, and equipped with skis.....
(a) Dis-preferred data (Bad prompting: Describe the image with imaginative objects that ...) ..... In the distance, group of trees stands tall. their branches heavy with snow. ..... Adding to the charm of the scene are two small, fluffy clouds that float in the sky, their softness providing gentle counterpoint to the skier's daring feat.
(b) Dis-preferred data (Normal prompting + Image corruption: Color jitter.)
The skier, clad in vibrant purple jacket and blue pants, The background is stark contrust to the skier, with a purple sky that adds sense of depth and dimension
(c) preeprefrd data (Normal prompting + Image corruption: Lower resolution.)
The image shows person engaged in a winter sport, likely skiing snowboarding, captured in mid-air against clear blue sky. The individual is wearing blue and yellow suit, .... The person is holding onto ski snowboard, which is also visible in the image. The motion blur effect
Figure 3: Examples of the self-constructed preference data in STIC.
following datasets. The recent development of LVLMs has significantly expanded the scale and diversity of VL cnstruction-folloiing data, including models such as LLaMA-Adapter V2 (Gao et al., 2023b), Qwen-VL. (Bai et al., 2023), InternVL. (Chen et al., 2023b), InstructBLIP (Dai et al., 2024), SPHINX-X (Gao et al., 2024). Claude-3 (Anthropic, 2024), MMI (McKinzie al., 2024), and Grok-1.5V (xAI, 2024). In this work, we focus on enhancing the visual perception and mathmatical reasoning capabilities of LVLMs by efficiently aligning them with purely unsupervised data.
Alignment fine-tuning. Subsequent to supervised fine-tuning (SFT), alignment fine-tuning has emerged as a prominent approach to further enhance the performance of LLMs by aligning them with human preferences (Ouyang et al., 2022; Casper et al., 2023). Early efforts utilized on-policy reinforcement learning (RL) methods, such as proximal policy optimization (PPO) (Schulman et al., 2017), to train a reward model based on preference data (Bai et al., 2022: Touvron et al., 2023a). With the notable introduction of direct policy optimization (DPO) (Rafailov al., 2023), a new line of research emphasizes direct learning from human preferences without relying on an explicit reward model (Zhao al., 2023; Azar al., 2024; Ethayarajh et al., 2024; Zheng et al., 2024). Another prominent direction is iterative preference fine-tuning, which has proven effective in enhancing model performance by repeatedly optimizing on newly generated preference pairs in each iteration (Adolphs et al., 2023; Xu et al., 2023; Xiong et al., 2023; Pang et al., 2024). While substantial research has focused on alignment fine-tuning for LLMs, efforts adapt these techniques for LVLMs have been significantly limited. Initial attempts involve constructing preference datasets using human-labeled data (Sun et al., 2023) or GPT-4 generations for fine-tuning with a DPO loss (Zhou al., 2024). Concurrent works (Pi et al., 2024: Zhou et al., 2024) begin to focus on generating preference dataset of LVLMs. while our method distinguishes itself with the unique preference prompt set. Self-training. Traditional et al., 2020; Zoph et al., 2020; Sohn et al., 2020; Ghiasi et al., 2021; Kang et al., 2023) leverage trained models to generate labels for unlabeled data and incorporate these self-labeled examples into training form of data augmentation. These frameworks primarily focus on self-supervised representation learning of vision models. While both classical self-training schemes and our approach share the fundamental goal of effectively utilizing unlabeled data to enhance model performance, our method differs in its focus on vision LLMs, maintaining an LLM as the backbone architecture. Rather than optimizing image representations, our approach aims generate synthetic data that enables the LLM to produce higher-quality responses to image queries.
superviedd training schemes (He et al., 2019; Xie et al., 2020; Wei
3 Problem Setting and Preliminaries
Notation. We use lower case letters and lower case bold face letters to denote scalars and vectors. We use the symbol p to represent the probability of LLM's response. And we denote the sequence of tokens generated from the LLM before the t-th token as y<r = [VI...., 3t-1] for t >>> 1.
Generative vision language models. LVLM typically consists of three components: vision encoder f()), a projection network g()), and an LLM po parameterized by 0. The model processes an image input e along with a text sequence [x1, [ Zn] as the prompt to generate a corresponding

---

response = lyi..... ..m,, where r; and yy represent individual tokens from the vocabulary of the LLM. The image is therefore converted into visual tokens within the language token space by the vision encoder and the projection network, producing v [V1,.... Fx. = fo g(e). The response then considered as sample from the conditional probability distribution po (, VV As a Markov process, the conditional probability distribution pe (y) v. x) can be decomposed as
m
po po x)
II po(u,lv,x,yes)) j=1
(3.1)
Alignment fine-tuning. To improve LLM alignment with human preferences, RL fine-tuning (Bai et al., 2022; Gao et al., 2023a) is typically employed after supervised fine-tuning (SFT). This process involves a reward function r(x. y) for a given sequence pair (x. y). The more preferred response y is expected to result in a higher reward y), where the corresponding objective is to maximize the following:
(( = Ex-
y)] XEx (X
(3.2)
where D is sampled from given distribution D and the KL regularization term prevents the new model po from deviating too much from the reference model Pref, with À > 0 as the regularization parameter. Training the reward function challenging in practice, but direct preference optimization (DPO) (Rafailov et al., 2023) simplifies this process using a predefined preference
dataset Sprct = {xx() Y Vi ()) )) iEINT' where y denotes the preferred response and y denotes the dispreferred response given the same prompt x The objective function then formulated as po(y))x)
(i)
e (Alog po(yulx) Allog pewe (vi(x) )
LDPO(O, Oret) = oo
(3.3)
(o..: xx)
where E(t) = logg(1 + (t) is the logistic loss function and Ozet is the reference model.
Our Method: STIC
In this section, introduce STIC, two-stage
algorithm designed to enhance image comprehension capabilities. The first stage constructs its own preference dataset and the second stage infuses the used SFT data with self- general framework of STIC. Notably, unlike recent work on fine-tuning algorithms (Sun et al., 2023; Zhou et al., 2024), STIC enables a base LVLM, such as LLa 1.6 (Liu et al., 2024), evolve from self-genen generatdd image captions, thus eliminating the need for additional supervised and preference data from human annotators or advanced teacher models. This approach fundamentally enhances image comprehension abilities and can be seamlessly applied to a wide range of vision-language reasoning tasks. We summarize STIC in Algorithms and and detail the process below.
image descriptions for fine-tuning. Figure 2 presents the
Stage 1:
Image comprehension self-training. The process begins with a self-constructed preference dataset from the base LVLM, which we aim to improve through fine-tuning. The dataset contains paired preference data for image descriptions:
Preferred response: Model-generated image descriptions derived from well-crafted prompts with explicit reasoning steps.
Dispreferred response: Model-generated descriptions resulting from either (1) corrupted image with low resolution or distorted color, or (2) "bad" prompts that cause the base model to hallucinate and describe elements that may not logically exist in the image.
The self-constructed preference dataset is used for the first-stage self-training using DPO (Rafailov et al., 2023) with an additional regularization term to further emphasize the preferred response, controlled by the hyperparameter The regularized loss function is follows:
L(O, rret) = Exxyyr))>s e Aog (y(y Alog
Pe.. Po... (y(|x) )
Po (o(y
a log pe (y (4.1)
Po,.. (y (x)
The use of an explicit loss term for positive examples can be similarly found in previous studies on contrastive learning (Chen et al., 2021; Chen and He, 2021; Chen et al., 2023a) and more recently in preference fine-tuning (Pang et al., 2024). Specifically, Chen et al. (2023a) demonstrated in the context of contrastive learning that regularization term applied to positive samples provably enhances the model's 's ability to differentiate between positive and negative samples. As demonstrated in our experiments in Section 6, the LVLM after Stage has shown notable improvement in downstream
5

---

Algorithm STIC (Stage I: image comprehension self-training)
Input: Unlabeled image dataset: (v(0) NE(N]- Image captioning prompt set: P= (x(*) he(M))
Hallucination prompt Phallu (x bally he|MM]- Image corruption h(-). Well-curated caption-
ing prompt: Xg. LVLM parameterized by 0:: Pe.
Let self-training dataset D = 0.
for i i I...N do
Randomly sample number n E (0,
Randomly sample x { hE[M].
Generate preferred response Yg ~ poo! (.1v((), x
if n < 0.5 then
Randomly sample bad prompt X ~ Phallu-
Generate dispreferred response yo ~ po. (- /v xx))
else
Corrupt the image input vi = h((((1).
Generate dispreferred response yb poo (-1vg)) ,x)
end if'
Add (x. Yb) D.
end for
no (y( > log po()s)x) Update 0 = argmingge EixyowSED (Alog pay (X,
a log ( (y, x)
po, (x)l)))
Output: 0,.
vision-language reasoning tasks, confirming that the enhanced visual comprehension ability directly benefits the model performance and its multimodal reasoning ability.
Prompt design. Our prompt design for the well-crafted prompt focuses on quality and diversity. We use GPT-4 to generate and sample multiple initial prompts, which are then refined through human filtering. To ensure effectiveness, we test these prompts on MSCOCO samples, verifying their ability to produce well-structured and relevant responses from the model. The bad prompts are sampled from GPT-4 and, in contrast, designed to elicit inaccurate descriptions by setting up slightly different task (describe objects that would logically exist in the image) for the model. We thus work under the assumption that responses generated from prompts that have differences in human preference lead to responses of the same preference with high probability. The key is that the discrepancy between good and bad prompts should result in pairs of responses that share the same implicit preference with high probability, which is sufficient for effective DPO training.
Stage 2: Description-infuser fine-tuning, In the second stage, we further fine-tune the self- eef-trained L.VM to leverage self- self-generated high-quality image descriptions for foslluitionfoolooiing tasks, and thus help ground its reasoning ability on self-generate descriptions. To achieve this, we randomly select a small subset of data from the model's instruction fine-tuning dataset already used during SFT. We then infuse the instructions in this subset with model-generated image descriptions as follows:
Image description: {model description)
coriginal instruction>
The original ground-truth truth completions remain unchanged. We then fine-tune the LVLM for one epoch on this small description-infusee infuusd subset. This fine-tuning step ensures that the model effectively integrates visual information into its responses, thereby enhancing its ability to handle variety of vision- langun-language reasoning tasks.
Describe and Respond. During inference, optionally, we can let the model self-augment its prompt for downstream vision-language reasoning tasks by describing the image before answering. Rather than generating an immediate response, we first elicit an image description, which is then concatenated with the original question produce more informed answer.
Experiments
In this section, we present the experiment results of STIC across seven visual question answering (VQA) benchmarks. We demonstrate that STIC effectively and substantially improves LVL.M' performance across different VQA tasks using self-constructed preference dataset without labels.

---

Algorithm 2 STIC (Stage 2: description-infused fine-tuning)
Input: Instruction-followigg dataset already used for fine-tuning the target LVLM model:
(v()), xx(i) y( hi(m] Image description prompt set: P {x des ME(M)) LVLM parameterized by 0 after self-training: po,.
(0)
Let description-infused dataset Daes f}-
for i = 1,... m do
Randomly sample Xdes (XeesiielM)
Generate model image description ydes ~ po. (-1v/(), xdes).
Add (Iydes, x, to Dacs. end for
Update 0 = argmingge E(xx)eDa ( Plog
Output: 0.
5.1 Experiment Setup
Model and datasets. In experiments, we consider 1lava-v1. 1lava-v1.--mistral--7 6-mistral-7b (Liu et al., 2023a) as our base model for self-training with model generated preference data. We additionally consider 1lava-v1.5- 7b (Liu et al., 2023a) based on Vicuna-7B (Chiang al., 2023b) directly compare with one concurrent baseline POVID (Zhou et al., 2024). A detailed discussion with POVID can be found in Appendix C.3. We follow the optimization process described in Section 4 for self- training on image description in Algorithm and description-intee infused finetuning in Algorithm to achieve improved downstream performances. For the self-constructed preference dataset, gather 66 unlabeled image data randomly sampled from the MSCOCO dataset (Lin et al., 2014) and specifically the train2014 split for its high-quality images popularly used for pre-training and fine-tuning. In the second stage, we randomly subsample 5k: used instruction fine-tuning data from LLaVA's SFT data construct the descriptioo-infusee fine-tuning data with model-generated image descriptions. Lastly, we use low-rank adaptation (LoRA) fine-tuning (Hu et al., 2021) instead of full fine-tuning for efficient computation. We defer the detailed prompts and corruptions Appendix B. Evaluation. We consider the widely used benchmarks for LVLM evaluation across different domains including: ScienceQA (Lu et al., 2022), TextVQA (Singh et al., 2019). ChartQA (Masty et al., 2022), LLaVA-Bench (Liu et al., 2023a), MMBench (Liu et al., 2023c). MM- Vet (Yu et al., 2023), and Math Vista (Lu al., 2024). Specifically, ScienceQA focuses scientific question answering and Math Vista focuses on math reasoning with visual information. TextVQA consists of images with text-rich contents and ChartQA with visual charts. Lastly, LLaVA-Bench. MMBench, and MM-Vet are three recent benchmarks to comprehensively evaluate a model's capabilities in a wide range of tasks and evaluation criteria. We use the evaluation scripts provided by LLaVA (Liu al., 2023a) to obtain the results for both our base model and after using STIC to ensure a fair comparison.
5.2 Main Results
Table I: Performance of STIC compared with the original LVLM model across vision-language reasoning tasks. For LLaVA-v1.5 (Vicuna 7B), we directly report the values in the paper of POVID, and " indicates an unreported value.
Model
ScienceQA
TexVVQa
ChanQA
LLaVA-Bench
MMBench
MM-Vet
Math
InstructRI.P (7R)
60.5
50.1
60.9
36.0
26.2
25.3
mPLUG-OWL2 (7B)
64.5
54.3
59.9
64.5
36.2
22.2
LLaVA-v1.5 (7B)
66.8
58.2
65.4
31.1
25.1
POVID
68.8
68.7
31.8
STIC
69.5
6.6
68.9
65.3
32.6
27.2
LLaVA-v1.6 (7B)
68.9
60.3
36.4
77.3
63.7
42.2
34.6
w/ STIC
75.3
41.5
79.2
45.0
37.0
We present our main results in Table and detail the benchmark performances of STIC (LLaVA-v1.,
7B) MMBench and MM-VE Vet in Figure 4. In Appendix B, we present detailed results for MMBench Table 7 and MM-Ver Vet Table 8. Our results show a consistent and significant improvement STIC over the original models (A-v11- and I.LVVA-v1.6 across all seven datasets. This improvement achieved using only self-constructed preference data and a small portion of the model's SFT dataset, which had already been used for fine-tuning the original model.
7

---

On average, STIC improves LLaVA-v1.5 by 1.7%. increasing from 45.3% to 47.0%. and LLa by a notable score 4.0%, increasing from 54.7% to 58.7%. The improvement comprehensive, as detailed in Tables 7 and 8, where STIC consistently enhances performance across all evaluation tasks and targets. Moreover, while STIC improves both I.LVA-- and LLaVA-vl a more significant improvement is observed in the more advanced model, LLaVA-v1.6 1.6.. This trend suggests that the extent self-improvement could be correlated with the model'ss inherent capabilities.
Original
Original
STIC
90
LR
gen
67.8 8
42
45.7
70-
math
FP-S
68.6
64.5
49.0
60.8
ocr
42.5
10.6
29.6
44.7
15.4
22.7
60
65.8
80,7
30.1
45,1
50
69.3
83.1
AR
CP
know
Spat
40
Complex
Detail
Figure 4: Accuracy improvement of STIC compared to the base LLaVA-v1.6 model across different tasks in Left: MMBench, where the original performances are re-scaled to 60 in plotting and STIC accordingly with the same coefficient for each task. Middle: MM- Vet, where the performances of the original model re-scaled 40 and STIC accordingly. Right: V.LaVA-Bench, where report the error bars over three independent runs due to the randomness of GPT-4 evaluation.
6 Ablation Studies and Discussions
In this section, we conduct ablation studies on the key components of STIC demonstrate their importance and effectiveness. Additionally, we examine the image distribution of our self-training data (MSCOCO) alongside the image distributions of benchmark datasets, revealing a positive correlation between performance gains and similarity in image distributions.
Effectiveness of describe-and tuning process in STIC by comparing it to the approach of directly allowing the base LVLM to describe image and then respond to query with self-augmented prompt, which refer as the snscribe-andrrespee
(DaR) prompting. We assess the significance of the fine-
(DaR) prompting method. As indicated in Table 2, applying DaR to the base LVLM yields mixed results, with performance improvements on some datasets and degradation on others, resulting in an overall average drop of 2.3%. In contrast, when DaR is combined with the fine-tuning process of STIC, it leads further average enhancement of 1.1% and notable 2.8% on ScienceQA. This demonstrates the synergistic effect of DaR and the fine-tuning process in STIC. Additionally, it worth noting that STIC achieves substantial average improvement of 2.8% even without the DaR prompting method, compared to the base LVLM.
Table 2: Test performance of STIC based on 11 llava-vi. wwmmsttall-aaaammmmmeemeaemeeere 7b. We investigate the benefit of DaR as a prompting method toward the base LVLM model as compared to the benefit on STIC.
Method
SuienceQA
TexiVQa
ChariQA
LLaVA-Bench
MMBench
MM-Vet
MathVista
Average
68.9
60.3
36.4
77.3
63.7
42.2
34.6
54.8
Original
69.9
56.6
34.6
78.5
50.7
42.3
34.7
52.5
72.5
63.4
39.3
78.4
68.7
45.7
35.2
$7.6
w/ STIC
75.3
65.2
41.5
79.2
67.8
45.2
37.0
58.7
Progression of stages. In Figure 5, we illustrate the sequential improvement in performance of STIC on ScienceQA. While stage focuses exclusively on enhancing the perception capabilities of the LVLM, it still notably improves performance on downstream VQA tasks. Building on the improved image comprehension achieved in stage I, stage introduces an enhanced reasoning process that utilizes the model's celf-generated image descriptions and results an even more significant gain. This enhancement further enables the model to self-augment its prompts with Describe and Respond (DaR), resulting total the substantial performance gains of 6.4% observed.
72.5
68.9
Sage
9age
D6R
Figure 5: Progression of stages in STIC.

---

40
20
20
ScienceQA TextvQA
60
MathVista ChartQA
-80
-40
80
Figure 7: 1-SNE visualization of images from MSCOCO and four benchmarks, each sampling kk
The role of dispreferred samples in STIC. To understand the importance of dispreferred samples in STIC, conduct ablation study using llava-vi. wewvavv...miitree..
the base LVLM. We remove the negative examples from the preference data and only use the positive samples for supervised fine-tuning (SFT), effectively creating an SFT version of STIC. Table shows that omitting the dispreferred samples even leads to a performance drop of 0.6% on LLaVA-Bench, while failing to provide equally significant improvement STIC with preference data. This highlights the crucial role of negative examples in aligning preferences and enabling the model to distinguish between high-quality and low-quality responses. By leveraging both positive and negative examples, STIC effectively improves the model's performance and generates more preferred outputs.
Table Test performance of STIC if we remove negative examples and use positive ones to perform SFT in Stage 1.
Model
ScienceQA
TextVQA
LLaVA Bench
Original
68.9
60.3
77.3
STIC (positive)
71.8
w/ STIC
75.3
65.2
79.2
Scaling law of STIC. We explore the scaling law of STIC by expanding the preference data in Stage 1. Using the LLaVA-Bench benchmark case study, scale up the preference data from 6k to 12k MSCOCO images. As depicted in Figure 6, there is an obvious gain on the LLaVA-Bench from 1.9% to 3.1%. This finding demonstrates that STIC can effectively leverage larger amounts of unlabeled image data and presents cost-effective solution the challenge of acquiring high-quality dision-language data.
2.5
19
3.0
Correlation between image distribution and performance gains.
To gain further insight into the effectiveness of STIC across different Figure 6: Scaling law in STIC. benchmarks, conducted t-SNE visualization analysis comparing the image distributions of MSCOCO, which we used for preference data construction, with those of four benchmarks: Sci- enceQA, TextVQA, Math Vista, and ChartQA (Figure 7). Our analysis revealed general trend: the greater the overlap between the MSCOCO image distribution and that of a benchmark, the higher the performance gain achieved by STIC on that benchmark. This observation held true for ScienceQA and TexIVQA, which exhibited substantial distributional overlap with MSCOCO and yielded the highest performance gains of 6.4% and 4.9 4.9%, respectively. Conversely, Math Vista, with its diverse image types and limited overlap with MSCOCO, saw a more modest gain of 2.4%. Interestingly, ChartQA was an outlier, achieving a high gain of 5.1% despite minimal overlap with MSCOCÓ, suggesting that the improved image comprehension from STIC played a fundamental role in under- standing and reasoning about the charts. Detailed per-benchmark visualizations and discussions are provided in Appendix C.2.
Diversity in image distribution. Based the observation the effect of image distribution in the final performance, we further utilize the Vision Flan dataset (VFLANT) for stage image comprehension self-training. This dataset includes images from 191 diverse vision tasks, providing broader spectrum of image types. We ensured a fair comparison by maintaining the same sample
lattps://hugginggaccccco http:///wwgagaaeeeooooo////aaaaa
flan 191- ttak__kk

---

Table 4: Performance of STIC on different stage-1 I training images compared with the original LVLM model LLa (Mistral 7B) across vision-language reasoning benchmarks.
LLaVA-Bench
MM-Vet
MMBench
Model
Data
Complex
Conv
Detail
All
Roc
Oer
Know
Gen
Spot
Math
Total
All
LLaVA-v1.6 (7B)
87.4
61.3
77.8
71.3
43.1
40.6
29.6
32,5
44.7
15.4
42.2
63.7
STIC
COCO
89.1
63.7
79.5
79.2
45.7
42.5
30.4
34,9
22.7
45.0
67.8
w/ STIC
VELAN
92.8
68.4
77.9
81.9
45.7
43.0
31.0
36.2
45.1
26.5
45.1
68.3
Table 5: Performance of STIC compared with the original LVLM model LLaVA-v1.6 (Vicuna 13B)
across assionlanguage reasoning tasks. Image data used for 13B model remain the same as what we used for the 7B model.
LLaVh-Bench
MM
MMBench
Model
Complex
Coev
Detail
All
Roc
Oct
Know
Gen
Spot
Math
Total
(178
87.4
61.3
77.8
77.3
43.1
40.6
29.6
32,5
44,7
15.4
42.2
63.7
6
94.0
73.8
78.7
84.5
52.2
47.1
38.8
45.2
42.7
26.9
48.9
70.6
w/ W/STIC
93.5
(8.1
79.4
85.6,-1.1)
54.5
48.0
42.3,43.5)
49.4
42.0
23.1
(0.5
72.
size (randomly sampled 64 images) and present the experimental results in Table 4. As shown, our approach improves consistently across different datasets, demonstrating its robustness and adaptability, Notably, the increased diversity of VFLAN led to further improvements in STIC, suggesting the potential for even greater enhancement with better sets of unlabeled images.
Scalability. To explore STIC's applicability to models with higher representation capacity, we conducted supplementary experiments using LLaVA-v1.6 1 (Vicuna-13B). Table 5 shows the detailed experiment results. We used the same images for STIC fine-tuning as our experiments for LLa v1.6 (Mistral-7B) to ensure fairness and the same set of hyperparameters. The improvements observed with LLaVA-vI. (Vicuma-13) 13B) demonstrate that STIC is not only effective with smaller models but also scales well with larger more capable LVLMs.
Qualitative example. In Figure 8, we show an example output of STIC. Despite the task being focused on mathematical reasoning, STIC enhanced the model's 's response by improving its image comprehension capabilities. While the original model merely identified one of the recognized numbers in the image as the final answer, the STIC fine-tuned model was able to interpret the meaning of cach number, describe them accurately, and perform reasoning based on this understanding.
What is the difference in metric tonnes between the amount of plastic produced and the amount discarded?
Query:
Plastic Waste Out of Control
In 2015 Strce out cirost es eAch plattic do Biy rado:
Base (LLaVV-vv.6 7B):
Fistciat
The difference between the amount of plastic produced and the amount discarded is 302 million metrie tonnes.
-DM
17M
STIC (L. VA 7B):
The image shows that 2015, humans produced 407 million
Packpts
metric of new plastic, while 302 million metric tons discarded as waste. The difference between the amount of plastic
produced and the amount discarded is 105 million metrie tons. This indicates that significant portion of the plastic produced is not being properly managed and is ending in the environment or landfills.
Fowe mas dennc ne Mearolle drcanie
Figure& 8: Response examples from original LLaVA-v1.6 and STIC (LLaVA-v1.6) in MM-
Conclusion
We introduce Self-Training on Image Comprehension (STIC), novel self-training approach designed to enhance the image comprehension capabilities of large vision language models (LVLMss. Our method leverages two-stage self-training process, creating preference dataset for image descrip- tions from unlabeled images and refining reasoning abilities through description-inie infused fine- tuning. Extensive experiments across seven vision-language benchmarks demonstrated significant perfor- mance improvements, with an average accuracy gain of 4.0%, while reducing the need for supervised fine-tuning data by 70%. Our findings underscore the potential of STIC to hamess vast quantities of unlabeled images, offering a cost-effective solution for advancing LVLMs
The promising results demonstrated by STIC in enhancing the capabilities of 7B LVLMs suggest its potential applicability to larger models, such as those with 40B, and 100B parameters, if computational resources permit.
10

---

Contribution Statement
The project began in January 2024 with Deng part of Gu's group, collaborating with Lu from Chang's ss group. This project was in the line of research on self-training using synthetic data developed by Gu and Deng. Deng proposed the initial idea of this project, which was further developed with Lu. In February 2024, Yin joined the project. Both Gu and Chang offered early feedback to help shape the project. In March, Chang met with Deng, Lu, and Yin discuss the initial experiment results and finalize the research plan. Chang continued supervising the project through one-on-one meetings with Lu. In April, Deng changed advisors and moved to Wang's 's lab, inviting Hu, Shen, Wang, and Zou to join the project. Hu and Shen contributed algorithmic improvements. while Wang and Zou provided detailed feedback on the experimental design and ablation studies. Deng and Lu primarily conducted the experiments and drafted the paper.
Acknowledgments
We sincerely thank the anonymous reviewers for their helpful comments. Pan is supported by Bloomberg Data Science Ph.D. Fellowship. Qualcomm Innovation Fellowship and UCLA Dissertation Year Fellowship. Fan and Kai Wei are supported by ONR grant N00014-23- 1-27800 OptumLabs, and CISCO gift grants. Wei is supported by DARPA HR001 1-24-9-0377, NSF 2200274, 2106859, 2312501, and NIH U54HG012517, U24DK097771. This paper is partially supported by Chan Zuckerherg Initiative. The views and conclusions contained in this paper are those of the authors and should not be interpreted as representing any funding agencies.
References
ADOLPHS,, I.., GAO, T., XU, J., SHUSTER, K., SUKHBAATAR, S. and WESTON, J. (2023). The cringe loss: Learning what language not to model. In Proceedings of the 61 Annual Meeting of the Association for Computational Linguistics (Volume 1: Longg Papers).
ALAYRAC, J.-B., DONAHUE, J., LUC, P., MIECH, A., BARR, I., HASSON, Y., LENC, K., MENSCH, A., MILLICAN, K., REYNOLDS, M. ET AL. (2022). Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems.
ANTHROPIC (2024). The claude 3 model family: Opus, sonnet, haiku.
AZAR, M. G., GUO, Z. D., PIOT, B., MUNOS, R., ROWLAND, M., VALKO, M. and CALAN- DRIELLO, D. (2024). A general theoretical paradigm to understand learning from human prefer- ences. In International Conference Artificial Intelligence and Statistics. PMLR.
BAI J., BAI, S., YANG, S., WANG, S., TAN, S., WANG, P., LIN, J., ZnoU, C. and ZhOOU, J. (2023). Qwen-vl: A versatile dision-langusal model for understanding, localization, text reading, and beyond. arXiv preprint arXiv:2.08. 12966
BAI, Y., JONES, A., NDOUSSE, K., ASKELL, A., CHEN, A., DASSARMA, N., DRAIN, D., FORT, S., GANGULI, D., HENIGIIAN, T. ET AL. (2022). Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862.
BAO. H., WANG, W., DONG, L., LIU, Q., MOHAMMED, O. K., AGGARWAL, K., SOM. S., PIAO, S. and WEI, F. (2022). Vlmo: Unified vision-language pre-training with mixture-of-moot wtteeeoooooomlll--eeeeeeeeeeeeee In Advances in Neural Information Processing Systems.
CASPER, S., DAVIES, X., SHI, C., GILBERT, T. K., SCHEURER, J., RANDO, J., FREEDMAN, R.. KORBAK, T.. LINDNER, D., FREIRE, ET AL. (2023). Open problems and fundamental limitations of reinforcement learning from human feedback. arXiv preprint arXiv:2307.15217.
CHEN, S., NIU, G., GONG, C., LI, J., YANG, J. and SUGIYAMA, M. (2021). Large-margin contrastive learning with distance polarization regularizer. In International Conference on Machine Learning. PMLR.
CHEN, X. and HE, K. (2021). Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition.
CHEN, X., WANG, X., CHANGPINYO, S., PIERGIOVANNI, A., PADLEWSKI, P., SALZ, D., GOOD- MAN, S., GRYCNER, A., MUSTAFA, B., BEYER, L. ET AT.. (2022). Pali: A jointly-scaled multilingual language-image model. arXiv preprint 2200:22009906994.
11

---

CHEN, Z., DENG, Y., LI, Y. and GU, Q. (2023a). Understanding transferable representation learning and zero-shot transfer in clip. arXiv preprint arXiv:2310.00927.
CHIEN, Z., DENG, Y., YUAN, H., JI, K. and GU, Q. (2024). Self-play fine-tuning converts weak language models to strong language models. arXiv preprint arXiv:2401.01335.
CHEN, Z., WU, J., WANG, W., SU, W., CHEN, G., XING, S., MUYAN, Z., ZHANG, Q., ZHU, X., LU, L. ET AL. (2023b). Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. arXiv preprint arXiv:2312.142258
CHIANG, W.-L.., LI, Z.,, LIN, Z.,, SHENG, Y., Wu, Z., ZHANG, H., ZHENG, L., ZHUANG, S., ZHUANG, Y., GONZALEZ, J. E., STOICA, I. and XING, E. P. (2023a). Vicuna: An open-source chatbot impressing gpt-4 with 90%* R chatgpt quality.
CHIANG, W.-L.., LI, Z.,, LIN, Z., SHENG, Y., WU, Z.,, ZHANG, H., ZHENG, L., ZHUANG, S., ZHUANG, Y.. GONZALEZ, J. E. ET AL. (2023b). Vicuna: An open-source chatbot impressing gpt-4 with 90%* * chatgpt quality. See https://vicuaa. Imsys. org (accessed 14 April 2023) 2 6.
DAI, W., LI, J., LI, D., TIONG, A. M. H., ZHAO, J., WANG, W., LI, B., FUNG, P. N. and Hoi, S. (2024). Instructblip: Towards general- purpose vision-language models with instruction tuning. Advances in Neural Information Processing Systems 36.
seneral-purpose
DENG, Y., ZHANG, W., CHEN,, Z. and GU, Q. (2023). Rephrase and respond: Let large language models ask better questions for themselves. arXiv preprint 2..iv:2311.042000
ETHAYARAJH, K., XU, W. MUENNIGHOFF, N., JURAFSKY, D. and KIELA, D. (2024). Kto: Model alignment as prospect theoretic optimization. arXiv preprint arXiv:2402.01306.
FRÄNKEN, J.-P., ZELIKMAN, E., RAFAILOV, R., GANDHI, K., GERSTENBERG, T. and GOODMAN, N. D. (2024). Self-supervised alignment with mutual information: Learning follow principles without preference labels. arXiv preprint arXiv:2404. 14313.
GAO, L., SCHULMAN, J. and HILTON, J. (2023a). Scaling laws for reward model overoptimization. In International Conference on Machine Learning. PMLR.
GAO, P., HAN, J., ZIAAGG, R., LIN, Z., GENG, S., Zhou, A., ZIIANG, W., LU, P., HE, C., YUE, X.. LI, H. and QIAO, Y. (2023b). Llama-adapter v2: Strameter-efficieet visual instruction model. arXiv preprint 30X1v:2304.15010
GAO, P., ZHANG, R., LIU, C., QIU, L.., HUANG, S., LIN, W., ZHAO, S., GENG, S., LIN, Z., JIN, P., ZHANG, K., SHAO, W., XU, C., HE, C., HE, J., SHAO, H., LU, P., LI, H. and QIAO, Y. (2024). Sphinx-x: Scaling data and parameters for a family of multi-modal large language models. In International Conference on Machine Learning (ICML).
GIIAASI, G., ZOPII, B., CUBUK, E. D., LE, Q. V. and LIN, T.-Y. (2021). Multi-task self-training for learning general representations. In Proceedings of the IEEE/CVE International Conference on Computer Vision.
GOEL, S., BANSAL H., BHATIA, S., ROSSI, R., VINAY, V. and GROVER, A. (2022). Cyclip: Cyelic contrastive language-image pretraining. Advances in Neural Information Processing Systems 35 67119.
HE, J., GU, J., SHEN, J. and RANZATO, M. (2019). Revisiting self-training for neural sequence generation. arXiv preprint 1rX.v:1909.13788
E. WALLIS, P., ALLEN-ZHU ZHU, Z., LI, Y., WANG, S., WANG, L., CHEN, W. ET AL. (2021). Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations.
JIA. C., YANG, Y., XIA, Y., CHEN, Y., PAREKH, Z., PHAM, H., LE, Q. V., SUNG, Y., LI, Z. and DUERIG, T. (2021a). Scaling up visual and langunne representation learning with noisy text supervision. In Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event (M. Meila and Zhang, eds.), vol. 139 of Proceedings of Machine Learning Research. PMLR
12

---

JIA, C., YANG, Y., XIA, Y., CHEN, Y.-T., PAREKH, Z., PHAM, H., LE, Q. SUNG, Y.-H., LI, Z. and DUERIG, T. (2021b). Scaling up visual and vision-language representation learning with noisy text supervision. In International Conference Machine Learning. PMLR.
JIANG, Q.. SABLAYROLLES, A., MENSCH, A., BAMFORD, C.,, CHAPLOT, D. S., CASAS, D. D. I.., BRESSAND, F., LENGYEL, G., LAMPLE, G., SAULNIER, L. ET AT. (2023). Mistral 7b. arXiv preprini urX1v:2310006825:
JOSIFOSKI, M., SAKOTA, M.. PEYRARD, M. and WEST, R. (2023). Exploiting asymmetry for synthetic training data generation: Synthie and the case of information extraction. arXiv preprint arXiv:2303.04132.
K KANG, G.-C., KIM, S., KIM, J.-H., KwAK, D. and ZHANG, B.-T. (2023). The dialog must go on: Improving visual dialog via generative self-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattenn Recognition.
KIM, D., PARK, C., KIM, S., LEE, W., SONG, W., KIM, Y., KIM, H., KIM, Y., LEE, H., KIM, AL. (2023). Solar 10.7 b: Scaling large language models with simple yet effective depth up-scaling. arXiv preprint arXiv:2312. 15166..
KIM, W., SON, and KIM, I. (2021). ViLT: ssion-and-languaee transformer without convolution or region supervision. In Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event (M. Meila and T. Zhang, eds.), vol. 139 of Proceedings of Machine Learning Research. PMLR.
LI, J., LI, D., SAVARESE, and Hor, S. (2023a). Blip-2: Bootstrapping language-image pre- training with frozen image encoders and large language models. In International conference machine learning. PMLR.
LI, J., LI, D., SAVARESE, S. and HoI, S. (2023b). Blip-2: Bootstrapping language-image pre- training with frozen image encoders and large language models. arXiv preprint arXiv:230..12597
LI, L. H., YATSKAR, M., YIN, D., HSIEI, C. and CHIANG, K. (2019). Visualbert: A simple and performant baseline for vision and language. CoRR abs/1908.03557.
LI, L. H., ZHANG, P., ZHANG, H., YANG, J., LI, C.,, ZHONG, Y., WANG, L., YUAN, L., ZHANG, L., HWANG, J.-N. ET AL. (2022). Grounded language-image pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.
LI, X., YIN, X., LI, C., ZIIANG, P., HU, X., ZIAAGG, L., WANG, L., HU, H., DONG, L., WEI, F., CHOL, Y. and GAO, (2020). Oscar: Object-semanties aligned pre-training for vision-language tasks. In Computer Vision ECCV 2020 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXX (A. Vedaldi, H. Bischof, Brox and Frahm, eds.), vol. 12375 of Lecture Notes in Computer Science. Springer.
LI, Y., BUBECK, S., ELDAN, R., GIORNO, A. D., GUNASEKAR, S. and LEE, Y. T. (2023c). Textbooks are all you need ii: phi-1.5 technical report.
LIN, T.-Y., MAIRE, M., BELONGIE, S., HAYS, J., PERONA, P., RAMANAN, D., DOLLÁR, and ZITNICK, C. L. (2014). Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13. Springer.
LIU, H., LI, C., LI, Y. and LEE, Y. J. (2023a). Improved basclines with visual instruction tuning. arXiv preprint arXhv:2310.03744.
LIU, H., LI, C., LI, Y., LI, B., ZHANG, Y., SHEN, S. and LEF, Y. J. (2024). Llava-next: Improved reasoning, ocr, and world knowledge.
LIU, H., LI, C., WU, Q. and LEE, J. (2023b). Visual instruction tuning. Advances in Neural Information Processing Systems (NeurIPS) 36.
13

---

LIU, Y., DUAN, H., ZHANG, Y., LI, B., ZHANG, S.. ZHAO. W., YUAN, Y., WANG, J., HE, C., LIU, Z. ET AL. (2023c). Mmbench: Is your multi-modke model an all-around player? arXiv preprint arXiv:3307.06281.
LU, P., BANSAL, H., XIA, T., LIU. J., LI, C., HAJIHHRZI, H., CHENG, H., CHANG, K.-W., GALLEY, M. and GAO, J. (2024). Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. International Conference on Learning Representations (ICLR).
LU, P., MISHRA, S., XIA, T., QIU, L., CHANG, K.-W., ZHU, S.-C., TAFJORD, O., CLARK, and KALYAN, A. (2022). Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems 35 2507-2521.
MASRY, A., Do. X. L., TAN, J. Q., JOTY, S. and HOQUE, E. (2022). Chartqa: A benchmark for question answering about charts with visual and logical reasoning. In Findings of the Association for Computational Linguistics: ACL 2022.
MCKINZIE, B., GAN, Z., FAUCONNIER, J.-P.. DODGE, S., ZHANG, B., DUFTER, P., SHAH, D.. DU, X., PENG, F., WEERS, F. ET AL. (2024). Mml: Methods, analysis & insights from multimodal Ilm pre-training. arXiv preprint 1rXXx:2403.096/1
OPENAI (2023a). Gpt-4 technical report.
OPENAI (2023b). Gpt-4v(ision) system card.
OUYANG, L., WU, J., JIANG, X., ALMEIDA, D., WAINWRIGHT, C., MISHKIN, P., ZHANG, C., AGARWAL, S., SLAMA, K., RAY, A. ET AL. (2022). Training language models to follow instructions with human feedback. Advances Neural Information Processing Systems 35 27730- 27744.
PANG, R. Y., YUAN, W., ClIO, K., HE, H., SUKIBATTAR, S. and WESTON, J. (2024). Iterative reasoning preference optimization. arXiv preprint 1XXv:22444.1933..
Pr, R., HAN, T., XIONG, W., ZHANG, J., LIU, R., PAN, R. and ZHANG, T. (2024). Strengthening multimodal large language model with bootstrapped preference optimization. arXiv preprint arXiv:2403.08730.
PRASAD, A., STENGEL-ESKKNN E. and BANSAL, M. (2023). Rephrase, augment, reason: Visual grounding of questions for vision-language models. arXiv preprint arXiv:2310.03861.
RADFORD, A., KIM, W., HALLACY, C., RAMESH, A., GOH, G., AGARWAL, S., SASTRY, G.,
ASKELL, A., MISHKIN, P., CLARK, J. ET AL. (2021). Learning transferable visual models from natural language supervision. In International Conference on Machine Learning (ICML). PMLR.
RAFAILOV, R., SHARMA, A., MITCHELL, E., ERMON, S., MANNING, C. D. and FINN, C. (2023). Direct preference optimization: Your language model is secretly reward model. arXiv preprint 1.Xi:23005.18290
ROSSET, C., CHENG, C., MITRA, A., SANTACROCE, M., AWADALLAH, A. and XIE. T. (2024). Direct nash optimization: Teaching language models to self-improve with general preferences. arXiv preprint 4rXiv:240.03375.
SCHULMAN, J., WOLSKI, F., DHARIWAL, P., RADFORD, A. and KLIMOV, O. (2017). Proximal policy optimization algorithms. arXiv preprint arXiv: 170.00677
SHEN, S., LI, L. H.. TAN, H., BANSAL. M., ROHRBACH, A., CHANG, K.-W., YAO, Z. and KEUTZER, K. (2022). How much can clip benefit vision-and-languaee tasks? In ICLR.
SINGH, A., HU, R., GOSWAMI, V.. COUAIRON, G., GALUBA, W., ROHRBACH, M. and KIELA, D. (2021). FLAVA: A foundational language and vision alignment model. CoRR abs/2111.04482.
SINGH, A., NATARJAN, V., SHAH, M., JIANG, Y., CHEN, X., BATRA, D., PARIKH, D. and ROHRBACH, M. (2019). Towards vqa models that can read. Proceedings the Computer Vision and Pattern Recognition.
the IEEE Conference
14

---

SOHN, K., BERTHELOT, D., CARLINI, N., ZHANG, Z., ZHANG, H., RAFFEL, A., CUBUK, E. D., KURAKIN, A. and LI, C.-L. (2020). Fixmatch: Simplifying semi-supervised learning with consistency and confidence. Advances in neural information processing systems 33 596-608.
SUN, Z., SIIEN, S., CAO, S., LIU, H., LI, C., SHEN, Y., GAN, C., GUI, L.-Y.. WANG, Y.-X., YANG, Y. ET AL. (2023). Aligning large multimodal models with factually augmented rlhf. arXiv preprint ar arXhv:2309.14555
TAN, H. and BANSAL, M. (2019). LXMERT: Learning cross-modality encoder representations from transformers. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019 (K. Inui, J. Jiang, V. Ng and X. Wan, eds.). Association for Computational Linguisties.
TAORI, R., GULRAJANI, I., ZHANG, T., DUBOIS, Y., LI, X., GUESTRIN, C., LIANG, P. and HASHIMOTO, T. B. (2023). Stanford alpaca: An fnstractionnfollooinng llama model.
TEAM, G., ANIL, R., BORGEAUD, S., WU, Y., ALAYRAC, J.-B., YU, J., SORICUT, R., SCHALK- WYK, J., DAI, A. M., HAUTU, A. ET AL. (2023). Gemini: A family of highly capable multimodal models. arXiv preprint arXhv:2312.11800
TOUVRON, H., MARTIN, L., STONE, K., ALBERT, P., ALMAHAIRI, A., BABAEI, Y., BASHLYKOV, N., BATRA, S., BIARGAVA, P., BIOOSALE, S. ET AL. (2023a). Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288
TOUVRON, H., MARTIN, L., STONE, K., ALBERT, P., ALMAHAIRI, A., BABAEI, Y., BASHLYKOV, N., BATRA, S., BHARGAVA, P., BHOSALE, S. ET AL. (2023b). Llama 2: Open foundation and fine-tuned chat models. arXiv preprint 23Xx:23307,09288
WANG, P., YANG, A., MEN, R., LIN, J., BAI, S., LI, Z., MA. J., ZHOU, C., ZHOU, J. and YANG, H. (2022a). Unifying architectures, tasks, and modalities through simple sequence-to-ssequeee learning framework. CoRR abs/2202.03052.
WANG, Z., YU, J., YU, A. W., DAI, Z., TSVETKOV, Y. and CAO, Y. (2022b). SimVLM: Simple visual language model pretraining with weak supervision. In ICLR.
WEL, C., SHEN, K., CHEN, Y. and MA, (2020). Theoretical analysis of self- self-training with deep networks on unlabeled data. arXiv preprint arXiv:2010.03622.
WU, T., YANG, G., LI, Z., ZIINGG, K., LIU, Z., GUIBAS, L., LIN, D. and WETZSTEIN, G. (2024). Gpt-4v (ision) is a human-aligned evaluator for text-to-3d generation. arXiv preprint arXiv:2401.04092
XAI (2024). Grok-1.5 vision preview.
XIE, Q., LUONG, M.-T., HovY, E. and LE, Q. V. (2020). Self-training with noisy student improves imagenet classification. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition.
XIONG, W., DONG, H., YE. C., ZHONG, H., JIANG, N. and ZHANG, T. (2023). Gibbs sam- pling from human feedback: A provable kl-constrained framework for rlhf. arXiv preprint arXiv:2312. 1456.
XU, J., LEE, A., SUKHBAATAR, S. and WESTON, J. (2023). Some things are more cringe than others: Preference optimization with the pairwise cringe loss. arXiv preprint 2rXi::2312.16622
YU, W., YANG, Z., LI, L., WANG, J., LIN, K., LIU, Z., WANG, X. and WANG, L. (2023). Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint arXiv:2308.02490.
YUAN, W., PANG, R. Y., CHO, K., SUKHBAATAR, S., XU, J. and WESTON, J. (2024). Self- rewarding language models. arXiv preprint arXiv:2401.10020
ZHAO, Y., JOSHI, R., LIU, T., KHALMAN, M., SALEH, M. and LIU, (2023). Slic-hf: Sequence likelihood calibration with human feedback. arXiv preprint XxXiv:2305.0422
15

---

ZHENG, C., WANG, Z., JI, H., HUANG, M. and PENG, N. (2024). Weak-to-strong extrapolation
expedites alignment. arXiv preprint arXiv:2404.16792.
Z Y., CUI, C., RAFAILOV, R., FINN, C. and YAO, H. (2024). Aligning modalities in vision large language models via preference fine-tuning, arXiv preprint arXiv:2402.11411
ZOPH, B., GHIASI, G., LIN, T.-Y., CuI, Y., LIU, H., CUBUK, E. D. and LE, Q. (2020). Rethinking pre-training and seainntain Advances in neural information processing systems 33 3833-3845.
16

---

A Additional Related Work
Self-improving language models. High-quality data, including human-crafted and advanced AT generated content, has been demonstrated to significantly enhance the performance L.LMs on various tasks (Josifoski al., 2023; Taori et al., 2023; Chiang et al., 2023a; Li et al., 2023c). Although, acquiring such high-quality data is often prohibitively expensive. To circumvent the costs associated with obtaining human-annotated or expertly curated data, researchers have shifted their focus to leveraging data generated by the target model itself, exploring ways of self-improvement (Chen et al., 2024; Yuan al., 2024; Fränken et al., 2024; Rosset et al., 2024). Recent studies have also emphasized the rephrasing capabilities of LLMs, which either enhance their own response quality (Deng et al., 2023; Prasad et al.. 2023) or augment synthetic data for sule-super fine-tuning (Kim al., 2023). To the best of our knowledge, our work is the first to explore the potential for seff-improvement in LVL.s, specifically focusing on the vision modality and emphasizing the self-improvement of image comprehension capabilities.
B Experimental Details
Perference data construction. We consider randomly sampling from the following "bad" prompts as a means of generating dis-preferred examples.
"Describe the image with imaginative objects that may exist in the scene."
"Enrich the description by adding hypothetical objects or characters that could be part the
scene."
"Suggest and detail practical items or people that could logically inhabit the image's setting."
"Incorporate elements that, though absent, would scamlessly fit into the context of the picture." "
"Imagine and describe additional everyday objects activities taking place just out of frame."
"Augment the scene with details of potential events items that plausible." "Conceive of and detail natural elements, such as weather or animals, that could realistically enter the scene. Make the description affirmative."
"Invent and incorporate details of practical tools, vehicles, or gadgets that could be expected in a similar scenario."
Given an input image, with 50% chance, we generate the dis-preferred response using the "bad" prompt. Otherwise, we generate the dis-preferred response with corrupted image either from color jittering or lower resolution. To generate the preferred response, we use the following step-by-step prompt:
"Please provide a detailed description of the image, focusing on the following. Identify the main subjects (people, animals, objects) in the image and describe what they are doing. Describe the setting of the image. Is it indoors or outdoors? What kind of environment or location does it depict? What mood does the image convey? Are there any specific elements (such as lighting, weather, expressions) that contribute to this atmosphere? Describe the dominant colors and the overall composition. How do these elements affect the image's impact? Point out any details or symbols that might be relevant to understanding the image's meaning or context. If applicable, provide interpretations of what the image might represent or communicate."
The prompts used as instructions in DPO are listed below:
"Illustrate the details of the picture." "Summarize the visual content presented."
"Explain what is depicted in the photograph."
"Outline the key elements captured in the image."
"Detail the composition and subjects within the frame."
"Covvey the atmosphere and mood represented in the snapshot."
'Interpeet the scene shown in the image."
"Identify and describe the main focal points in the visual."
In Stage 2, we prompt the LVLM with simple instructions like "Explain what depicted in the photograph." to gather the image descriptions for Stage 2 fine-tuning.
17

---

Task: Explain what depieted the photograph.
Preferred (Good prompting)
The image depicts social setting, likely dinner party gathering of some sort. The main subjects in the image bottle of St-Germain liqueur and several wine glasses. The bottle is prominently placed in the foreground, with its label facing the viewer, while the wine glasses scattered around the table, some filled with wine The setting appears to be indoors, possibly dining similar space, indicated by the presence of dining table. The table is covered with white tablecloth.....
ST-GERMAIN GEGHAIN
Dis-preferred (Bad prompting)
In the image, there's a sense of social gathering or a celebration. Just out of frame, you might imagine several every- day objects and activities!nnnl. A person is holding glass of wine, toasting to special occasion. in2, A waiter is walking by, carry ing tray with appetizers or desserts, In3. A guest is engaged in conversation with another person, sharing stories and laughter.n4. A person is laking pholograph of the momeni, capluring the joy and carnura- derie. im5, A host is standing near the bar, ensuring everyone has what they need.
Figure 9: Example of generated preference data, where the dis-preferred response is generated from bad prompting.
Task: Explain w what is depicted in the photograph
Preferred (Good prompting)
The image depicts a young child, likely boy. seated dining table with birthday cake in front of him. The child appears candles the cake, which
the process blowing and casting warm glow on face.
The child wearing a gray t-shirt and short, light-colored hair. ''hTThe setting indoors, with wooden dining table chairs visible in the back- ground. The environment suggests domestic, homey atmosphere, possibly dining room or kilchen area. ininThe mood of the image is joy
Dis- dis-preferred (Lower resolution)
The image shows close-up of person, likely child, looking at cake with candles. The cake appears to be birth- day cake, and the child seems focused it. The background is blurred, but it looks like an indoor setting with warm lighting. The focus the child the cake, making the background indistinct.
Figure 10: Example of generated preference data, where the dis dis- preferreed response is generated from images with lower resolution.
Task: Interpret scene shown the image.
Preferred (Good prompting)
The image depicts a joyful and festive scene, likely birthday party. The main subjects in the image celebration, and several other children and adults. The child is seated at table. engrossed in cating piece of cake. ..... The setting is indoors, with the table and chairs urrunged lypical party layout. The table is covered with table- cloth featuring cartoon character, udding to the festive atmosphere. The mood of the image
young child, who appears to be the focus of the
happiness and anticipation
Dis-preferred (Color jitter)
The imuge shows child's birthday party. The child the foreground is wearing striped shirt and appeurs to be in the process of cating a picce of cake. The cake is green and has a H'Hppy Birthday." message on it, suggesting it's birthday colehration. The child is using plastic fork to cat the cake. innnnn the background, there are other children and adults, some of whom are also eating cake. The overall atmosphere seems to be one of joy and celebration.
Figure 11:
Example of generated preference data, where the dis dis-preferred response is generated from images with color jittering.
Fine-tuning details. We train for epoch in each stage, including the image comprehension self- training stage and the deseription-infun LoRA mee-tunng in both stages, with lora_r r = 128, lora_alpha = 256, and lora_ lora_target = all. The fine-tuning hyperparameters for Stage are presented in Table 6. The parameters remain the same for Stage 2 fine-tuning, with the only differences being learning rate of 2e - 5 and a global batch size 64.
fine-tuning stage. We use the same hyperparameters for
18

---

Table 6: Fine-tume Fine-tuning hyperparameters.
Learning rate Optimizer
lc 7
Adamli
Global batch size
4
Regularization coefficient O
1/1024
prt_porooeeorrrr mm
2e
wtt_ppoojecto__yyppe
m mlp2x_gell
gradieent_acccmllaiio.....o
image_aspeect_ratio group_by. broup_by__oddlitt__lenggggg
pad True
weight_decay
0
varmup_ratio
0.03
1r sr_scheduleertyp _tpee
cosine
1024
Describe and Respond.
User: <image>\n Detail the composition and subjects within the frame.
Model:
<image description>
User: <image>\nImage description: An<image descriptiio>>l(<uustttiie
Model: <response>
Evaluation details. We use the evaluation scripts provided by LLaVA (Liu al., 2023a) for all evaluations. It is important to note that the potential new evaluation scripts and prompts used to report the results for LLaVA-v1.6 have not been released the time of writing this manuscript. This may cause discrepancies in the evaluation results of the original model.
Compute resources. Experiments were conducted on NVIDIA RTX A6000 GPU clusters. The entire self-training process for LLaVA v1.5 (7B) and LLaVA v1.6 (7B), using 6k image data and 5k: reused instruction fine-tuning data, takes approximately hours 4 GPUs. The evaluation process of STIC for the benchmarks typically varies from 2 to 8 hours, mainly depending on the test set size.
C Experimental Results
Table 7: Detailed performance of STIC com- pared with the original VLM model on the MM- Bench dev set.
Table 8: Detailed performance of STIC com- pared with the original VLM model on the MM- Vet benchmark.
MMBench
MM- Vet
Model
Model
I.R
AR
RR
FP-S
FP-C
CP
rec
know
gen
spat
math
Original
35.6
65.8
64.5
80.7
Original
43.1
40.6
32.5
44.7
15.4
STIC
42.4
69.3
67.8
68.6
60.8
83.1
w/ STIC
45.7
42.5
30.4
34.9
45.1
22.7
C.1
Example Outputs
In Figure 12 and 13, we demonstrate more output examples of the original LLaVA-v1.6 and STIC.
Query: am getting one funghi pizza one prosciutlo pizza. How much should pay in total?
WIFi FREE
PIZZA
Base (I...V--v.. 7B):
HARREÑTAA 2. Via.
CANICCIUAA
Based on the menu prices, the total cost for one funghi pizza one prosciutto pizza would 14,00E.
7 40c.
SURGLL 7 THe
STIC (A.v. 7B):
PRTACIOTC
Based the menu displayed in the image, the price for funghi
500 SOPSOUNLL
pizza 6,90C, and the price for prosciutto pizza is 7,90C. Therefore, the cost for one funghi pizza and one prosciutlo
CAL
BALZONE 14.00
pizza would be 14,80C.
Figure 12: Response examples from original LLa v1. and STIC (LLaVA-v1.6) in MM- Vet.
19

---

Query: What the name of man the photo?
Base (L.LaVA-vI.6 7B):
I'm sorry, but I cannot identify the person the photo.
STIC (LLaVA-v1.6 7B): The man in the photo is Elon Musk,
Figure 13: Response examples from original LLaVA-v1.6 and STIC (LLaVA-v1.6) in LLaVA-
C.2 t-SNE Visualization Analysis
ScienceQA, TextVQA, Math Vista, and ChartQA were chosen because there are at least 1, 000 images in the test set, providing enough data points for analysis.
MSCOCO vs ScieneeQA. The gain achieved by STIC on ScienceQA was 6.4%, the highest across all seven benchmarks. As evident from Figure 14 (a), the images in ScienceQA have substantial overlap with those in MSCOCO. This suggests that the image comprehension capabilities developed by STIC through self-training on MSCOCO translated effectively to the scientific reasoning tasks in ScienceQA.
MSCOCO vs TextVQA. STIC yielded gain of 4.9% on TextVQA, one of the higher gains observed across the benchmarks. Figure 14 (b) shows significant overlap between the image distributions of TextVQA and MSCOCO. This indicates that the enhanced image understanding from self-training on MSCOCO proved beneficial for the text-based visual question answering tasks in VeA.VQA
MSCOCO vs MathVista. On Math Vista, STIC achieved gain of 2.4%, which, while still notable, was lower compared to other benchmarks. As Figure Figure 14 (c) illustrates, the images in Math Vista have limited overlap with those in MSCOCO. Moreover, Math Vista features diverse image types and mathematical reasoning tasks that pose additional challenges beyond image comprehension. These factors likely contributed to the more modest performance gain.
MSCOCO vs ChartQA. STIC attained a high gain of 5.1% on ChartQA, despite the images in ChartQA having minimal overlap with those in MSCOCO, as shown in Figure 14 (d). This seem- ingly contradictory result can be explained by considering that the improved image comprehension capabilities developed by STIC play fundamental role understanding and reasoning about the charts in ChartQA. Thus, even with limited distributional similarity, the enhanced perception skills proved valuable for this benchmark.
C.3 Discussion with POVID.
We detail the differences between STIC and POVID. In POVID, the dispreferred response is generated either by adding Gaussian noise to the original image or by manually injecting hallucinations into the ground truth completion, using the labeled object information of the images. In contrast, STIC (I) specifically targets the image description task, (2) constructs preference datasets exclusively from unlabeled images using self- -efeeeeredted content for both preferred and dispreferred responses, (3) employs an automatic model generation process without manual injections or modifications, and (4) utilizes only small portion of SFT data for instruction-fon followinn fine-tuning with uniquely infused model descriptions. Lastly, we compare the data types and scales used in POVID and STIC in Figure 15.
Comparison of Data Volumes
Data Unlabeled Data
15
Data Volume (1k)
5
POVID
STIC
Figure 15: Data comparison.
C.4 Investigation of Prompt Quality
Table 9 presents an additional experiments using DaR to demonstrate prompt quality. We compared normal prompts from our main paper (e.g.. "Illustrate the details of the picture.") with the hallucination prompts and well-curated prompts used for DPO pair generation. The results show an expected diserepancy in QA performance: hallucination prompts significantly decreased performance, while well-curated prompts maintained decent performance. We also included results based prompt proposed by Llama-3 8B and filtered using the same restrictions, The performance difference between GPT-4 and Llama-3 8B prompts underscores the quality of GPT-4's proposals.
20

---

COCO
TextvQA
20
20
0
-60
COCO
440
-80
ScienceQA
80
-60
-40
-20
20
40
60
-80
-60
-40
-20
20
40
60
80
(a) MSCOCO and ScienceQA
(b) MSCOCO and TextVQA
60
40
0
20
-20
COCO
COCO
60
MathVista
-40
ChartQA
-60
-40
-0
20
-60
-0
80
(c) MSCOCO and Math Vista
(d) MSCOCO and ChartQA
Figure 14: t-SNE visualization of images from MSCOCO and benchmarks.
Table 9: Test performance of llava-v1. 6-mistral-78 using various prompts with DaR. We evaluate prompt quality using DaR as a prompting method. DaR=None represents the original LVLM model's performance. Normal prompt refers the simple prompt we used for DaR in our paper. GPT-4's well-curated prompt refers to the prompt we used for preferred response generation, and we include Mistral 7B's curated prompt for additional comparison.
Model
DaR
LLaVA-Bende
MM-Vet
MMBench
None
77.3
42.2
63.7
Normal Prompt
78.5
42.3 (20.1)
50.7(-13.0)
1LLVA-v1.6 (7B)
Hallu Prompt
73.7 (3.7
40.5 (0.5(-1.7)
40.7(-23.0)
Well-curated (I (lamm-3 8B)
77.2
40.0
60.1 (0.1(-3.6)
Well-curated (GPT-4)
79.1
42.9 (2.9
60.9(-2.8)
D Limitations and Future Work
While STIC demonstrates significant performance gains across diverse set benchmarks, there are still some limitations to be addressed in future work. First, STIC achieves relatively small accuracy gain on Math Vista compared to other benchmarks. This likely because Math Vista features various mathematical reasoning abilities across wide range of image types, from elementary school to college level problems, which go beyond the scope of image comprehension. In contrast, STIC uses MSCOCO, containing only natural images, for constructing its preference data. To further improve performance on tasks like Math Vista, future work could expand the source image types and generate task-specific image description data that better aligns with the target benchmark.
Second, while our approach of using good prompts for positive examples and bad prompts or corrupted images for negative examples is straightforward and effective, this preference data may be insufficient for scenarios requiring nuanced understanding of image details. More sophisticated strategies for preference data construction could potentially further boost fine-tuning performance with the DPO loss.
Additionally, our current method relies on a two-stage process of first training on the preference data and then performing Eescription-infusel fine-tuning. An interesting direction for future work would be to explore integrating these stages into a single end-to-end training process, which could potentially lead even greater synergies and performance gains.
Lastly, while we have demonstrated the scalability of STIC by doubling the amount of preference data, leading to further improvements, we have not yet explored the upper limits of this scaling. It is
21

---

possible that even larger amounts of self-training data could lead to diminishing returns at some point. Characterizing the scaling behavior of STIC more fully is an important direction for future research. Despite these limitations, we believe STIC represents an important step towards leveraging the vast amounts of unlabeled image data available to enhance the image comprehension capabilities of large Iasion-language models in a cost-effective manner.
E Broader Impacts
The development of STIC, our self-training approach for enhancing the image comprehension capabilities of large vision language models (LVLMs), presents several potential societal impacts. Positively, our method can democratize access to advanced vision-language models by significantly reducing the cost and effort required for fine-tuning, making state-of-the-art LVLMs more accessible to researchers and organizations with limited resources. This can accelerate advancements in healthcare, education, and environmental monitoring, where improved image comprehension can lead to better diagnostic tools, personalized learning experiences, and more effective environmental protection measures. Additionally, by encouraging the reuse and recycling of existing data, STIC aligns with sustainable AI practices, promoting efficient use of computational and data resources.
However, there are potential negative societal impacts that must be considered. Enhanced LVLM capabilities could be misused for generating disinformation, creating fake profiles, or conducting unauthorized surveillance. contributing the spread of misinformation and erosion of public trust. Fairness considerations are crucial, as biased training data may lead to outputs that disproportionately impact specific groups, resulting in unfair treatment or discrimination. Privacy concerns also arise from using self-generated data, particularly if models are trained on sensitive or personal visual content without proper consent. To mitigate these risks, strategies such as gated release of models, robust fairness audits, diverse data inclusion, and enhanced transparency about the technology's limitations and risks are essential to ensure responsible use.
22

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The main claims the abstract and introduction focus the novel self-training method for image comprehension, description-infusse fine-tuning, and the significant per- formance gains across several benchmarks. These contributions are accurately reflected and supported by the theoretical and experimental results presented in the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: The limitations of the proposed method are thoroughly discussed in Appendix D, where the authors address potential challenges and constraints, including domain align- ment issues and possible failure cases, providing clear reflection the robustness and scope their approach.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point any strong assumptions and how robust the results are violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
23

---

3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: Section provides the problem setting and preliminaries, outlining the as- sumptions and theoretical foundations supporting our proposed approach. Theoretical assumptions are clearly stated and supported by relevant literature.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Section 5 and the Appendix provide detailed information on the experimental setup, datasets, and evaluation metrics, ensuring reproducibility. Additionally, we commit to releasing the entire codebase, including data generation and fine-tuning procedures, to facilitate replication of our results.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
24

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
The paper commits to providing open access the code, including data generation and fine-tuning procedures. We discuss the open benchmarks used for evaluation in Section 5.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.c cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
/CodeSoomiisseeeeeee hwtrbbmmsssioooooiiooiooo sesssnnnnycyy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: Section provides details on the experimental setup, Section 6 covers the ablation studies, and Appendix B includes additional experimental details such as data splits, hyperparameters, and the type of optimizer used.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
25

---

Justification: On LLaVA-bench, where the evaluations are done with GPT-4, we reported the error bars in Figure However, most evaluations in this work are done using greedy decoding, resulting in deterministic process that does not incur error bars. Therefore, the reported values the baseline method that we compare to similarly did not report error bars. Lastly, due to the computational expense of fine-tuning large language models multiple times, we did not include error bars. However, we acknowledge this limitation and ensure that the reported results are consistent and reliable based on our experimental setup. Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots. The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Appendix B provides detailed information on compute resources used for experiments, including the type of compute workers (GPU) and execution time.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't it make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: / /tt//nnursp..// hccppullic/Ehhhccuuleeeeee innes??
Answer: [Yes]
The research conducted in the paper adheres to the NeurlPS Code of Ethics, ensuring ethical standards are maintained in data usage, experimental procedures, and reporting of results. The paper preserves anonymity and considers the implications of the research society.
Justification:
Guidelines:
26

---

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Appendix E discusses both the potential positive societal impacts, such as advancements in vision-languaan understanding and applications in education, and the po- tential negative societal impacts, including concerns about misuse and ethical considerations. The paper also addresses possible mitigation strategies for negative impacts.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymentt of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For cxample, it is legitimate to point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our approach does not involve releasing models or data, so there are no high-risk elements requiring safeguards.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
27

---

12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We use open-sourced models and datasets for our experiments, all of which are properly cited and used under their original licenses.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not introduce any new assets, so this question not applicable.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing or research with human subjects, so this question is not applicable.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
28

---

According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based on the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing research with human subjects, this question is not applicable.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
29


Reviews:
[{"review_id": "gR5QPe1Y0B", "reviewer": "Reviewer_qFQt", "summary": "This paper introduces Self-Training on Image Comprehension (STIC), which emphasizes a self-training approach specifically for image comprehension. First, the model self-constructs a preference dataset for image descriptions using unlabeled images. Preferred responses are generated through a step-by-step prompt, while dis-preferred responses are generated from either corrupted images or misleading prompts. To further self-improve reasoning on the extracted visual information, the model reuses a small portion of existing instruction-tuning data and appends its self-generated image descriptions to the prompts. Improvements in several benchmarks are reported.", "questions": "What if we try the proposed method with smaller or bigger models or other ViT families (e.g., EVA-CLIP models)? With a higher representational capacity, will the model benefit more or less from self-training?", "limitations": "The authors adequately addressed the limitations.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The challenges of self-training with VLM are discussed, which is appreciated. \n\n2. The proposed STIC approach is claimed to be a novel two-stage self-training method that targets both image perception and reasoning over images and texts, which is intriguing.\n\n3. STIC does not require pre-labeled information on the images,\n\n4. The methodology of constructing dis-preferred data using bad prompting is pretty interesting.", "weaknesses": "1. The experiments are conducted with 7B-level LLava 1.5 and 1.6. The method's scalability remains questionable.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your strong support and valuable feedback! We address your major comment as follows. \n\n### **W1: The scalability of STIC**\n\nTo explore STIC's applicability to models with higher representation capacity, we conducted supplementary experiments using LLaVA-v1.6 (Vicuna-13B). \n\n| **Model**        | **LLaVA-Bench** (Conv) | **LLaVA-Bench** (All) | **MM-Vet** (Gen) | **MM-Vet** (All) | **MMBench** |\n|------------------|-----------------|-----------------|-----------------|------------|-------------|\n| LLaVA-v1.6 (7B)  | 61.3 | 77.3         | 32.5   | 42.2       | 63.7        |\n| LLaVA-v1.6 (13B) | 73.8 | 84.5       | 45.2     | 48.9       | 70.6        |\n| LLaVA-v1.6 (13B) w/ STIC         | **78.1** | **85.6**     | **49.4**       | **50.5**       | **72.3**        |\n\nTable 1 in the [attached one-page pdf](https://openreview.net/attachment?id=VV3MK6FiS7&name=pdf) shows the detailed and comprehensive experiment results. Due to compute and time constraints, we included the three benchmarks (LLaVA-Bench, MM-Vet and MMBench) that include various different tasks and can comprehensively evaluate the model’s performance. We used the same images for STIC fine-tuning as in our experiments for LLaVA-v1.6 (Mistral-7B) to ensure fairness and the same set of hyperparameters due to time constraint. The improvements observed with LLaVA-v1.6 (Vicuna-13B) demonstrate that STIC is not only effective with smaller models but also scales well with larger or more capable LVLMs. It also shows potential for further improvement through hyperparameter tuning, data filtering, and enhanced data generation. \n\nWe hope that our additional experiments address your raised concerns. Let us know if there remain further questions, and we are happy to discuss them. \n\n### **Q1: What if we try the proposed method with different LVLM models?** \n\nThank you for this insightful question. In our original and additional experiments, we employed STIC with LLaVA-v1.5 and LLaVA-v1.6, incorporating various LLM backbones at different scales, specifically Vicuna-7B, Mistral-7B, and Vicuna-13B. These models, all incorporating strong visual encoders from the CLIP family, demonstrated effective improvements with STIC. While our current exploration of different models was constrained by time and computational resources, we recognize the importance and potential of exploring a wider range of LVLM models. Future work could investigate models with diverse architectures and LLM backbones, such as Llama-3, to further explore the potential of our proposed method.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1XWiiuMiAf", "reviewer": "Reviewer_B7oX", "summary": "This paper proposes a two-stage method to enhance Large Vision Language Models (LVLMs) using unlabeled images. In the first stage, well-designed good and bad prompts are used to make the LVLM generate preferred and dis-preferred completions, respectively, conditioned on the unlabeled images (from COCO). Then, direct preference optimization (DPO) is used to fine-tune the LVLM using the generated preferred and dis-preferred completions. In the second stage, the fine-tuned model is used to generate descriptions for images in an instruction-tuning dataset (from LLaVA's data). The generated descriptions are inserted into the instruction-tuning data to fine-tune the model. The fine-tuned model shows significant improvement over the baseline (the LVLM before fine-tuning) on seven VLM benchmarks.", "questions": "- What is the difference between the image captioning prompt set $P$ and the well-curated captioning prompt (Algorithm 1)?\n- What is the prompt $x$ used in DPO in stage 1?\n- What is the model performance if only stage 1 is performed (i.e., without stage 2's SFT)?", "limitations": "See Weaknesses.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The biggest strength is the significant improvement over the baseline LVLM achieved by the proposed method. There is an average improvement of 4 points on the seven VLM benchmarks. \n- The proposed method mainly leverages unlabeled images for training, which gives the proposed method a great potential to use a vast amount of unlabeled images. The authors also show using more unlabeled images for training improves the performance.", "weaknesses": "- The paper raises some questions unanswered.\n  - The effect of the prompt set is less explored in the paper. It seems that the prompts play a crucial role in data construction. It is unclear how the authors designed the well-curated captioning prompt and the hallucination prompt set. Are there any principles behind the design? Especially for the well-curated captioning prompt that generates the preferred data, how do different design choices affect the final model performance in the evaluation?\n  - Is the performance gain dependent on the MSCOCO data set? Do other image datasets (such as Flickr30k) work in stage 1? \n- It seems counterintuitive that fine-tuning an LVLM on MSCOCO will help improve its performance on science-related benchmarks like ScienceQA. It will help us better understand the mechanism by comparing the model generations in the benchmarks before and after using the proposed method. \n- The description of describe-and-respond (DaR) prompting is a bit unclear. I could not fully understand the setting.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your support and the constructive feedback that helped us improve our work. Please see our detailed response with additional experiments below. \n\n### **W1a: Principles behind the prompt design.**\n\nIn short, we use GPT-4 to generate and sample multiple initial prompts, which are then refined through human filtering. To ensure effectiveness, we test these prompts on MSCOCO samples, verifying their ability to produce well-structured and relevant responses from the model. Using DaR performance as an evaluation to the prompts (Table 2 of the [attached one-page pdf](https://openreview.net/attachment?id=VV3MK6FiS7&name=pdf)), we showed that the better crafted prompts result in better performance for DaR even on plain models. \n\nPlease see our detailed response in [Global A1 in the global rebuttal](https://openreview.net/forum?id=FZW7Ctyjm3&noteId=VV3MK6FiS7). We will include the discussion and experiments in our revision. \n\n### **W1b: Is the performance gain dependent on MSCOCO data?**\n\nThank you for raising this important point. To address this, we conducted additional experiments using images from various sources. Specifically, we utilized the Vision Flan dataset (VFLAN: https://huggingface.co/datasets/Vision-Flan/vision-flan_191-task_1k) for stage 1 image comprehension self-training. This dataset includes images from 191 diverse vision tasks, providing a broad spectrum of image types.\n\nWe ensured a fair comparison by maintaining the same sample size (randomly sampled 6,000 images) and have presented the experimental results in Table 3 of the [attached one-page pdf](https://openreview.net/attachment?id=VV3MK6FiS7&name=pdf). The results indicate that our approach improves consistently across different datasets, demonstrating its robustness and adaptability. Notably, the increased diversity of VFLAN led to further improvements in STIC, suggesting the potential for even greater enhancement with better sets of unlabeled images. This finding aligns with our analysis in Figure 8 of the main paper, where we observed a positive correlation between the overlap of MSCOCO's image distribution with a benchmark and the performance gains achieved by STIC on that benchmark.\n\n### **W2: It’s better to compare model generations before and after using STIC.**\n\nThank you very much for the suggestion. In Figure 1 of our submission, we showed an example of model generation before and after using STIC. In line 75-76, we provided a discussion that STIC improves the model response by successfully identifying the key visual information for subsequent reasoning. \n\nIn the [attached one-page pdf](https://openreview.net/attachment?id=VV3MK6FiS7&name=pdf), we included two additional model generation examples before and after applying STIC, as illustrated in Figures 2 and 3. Despite the task being focused on mathematical reasoning, STIC enhanced the model’s response by improving its image comprehension capabilities. While the original model merely identified one of the recognized numbers in the image as the final answer, the STIC fine-tuned model was able to interpret the meaning of each number, describe them accurately, and perform reasoning based on this understanding.\n\nFurthermore, in Figure 8 and its corresponding ablation study of our main paper, we examined the improvement in ScienceQA, where it shares a great overlap between the MSCOCO image distribution. \n\n### **W3: Further explanations on DaR prompting.**\n\nWe apologize for the lack of clarity in our initial presentation on DaR. Please see our explanation in [Global A3 of our global rebuttal](https://openreview.net/forum?id=FZW7Ctyjm3&noteId=VV3MK6FiS7). We will add a paragraph in our method section to fully illustrate DaR in the revision. \n\n### **Q1: Difference between the image captioning prompt set and the well-curated prompt.**\n\nWe detailed the image captioning prompt set in our answer to Q6. Here are the key differences between the image-captioning prompt and well-curated captioning prompt: \n\n**Image Captioning Prompt Set**: This set comprises concise and straightforward prompts designed to elicit basic image descriptions. These prompts typically ask the model to describe the image in simple terms without additional guidance or structure. \n\n- Purpose: The prompts in set P serve as the target task.\n\n**Well-Curated Prompt**: These prompts are designed to be more elaborate and structured, crafted to elicit higher-quality responses by encouraging the model to engage in a more systematic reasoning process.\n\n- Purpose: Generate superior responses to provide a learning signal for preferred/positive responses.\nThis process alone (without the negative/dispreferred responses) is similar to the currently popular method called system 2 distillation [1], where the model is fine-tuned on its responses generated from a more complex, step-by-step prompt. The goal is to teach the model to apply the enhanced reasoning patterns induced by the well-curated prompts when responding to the simpler prompts (e.g. in set P). \n\nWe will add the above explanation in our next revision. \n\n[1] Distilling System 2 into System 1\n\n### **Q2: What is the prompt 𝑥 used in DPO in stage 1?**\n\nWe included some of the prompts below due to character limit and will include the full set of eight prompts in our future revision. \n\n- \"Illustrate the details of the picture.\",\n- \"Summarize the visual content presented.\",\n- \"Explain what is depicted in the photograph.\",\n- …\n\n### **Q3: Model performance if only stage 1 is performed.**\n\nThank you for raising this important point. In short, while stage 1 focuses exclusively on enhancing the perception capabilities of LVLM, it still notably improves performance on downstream VQA tasks (1.1\\% accuracy gain on ScienceQA).\n\nPlease see our detailed response in [Global A2](https://openreview.net/forum?id=FZW7Ctyjm3&noteId=VV3MK6FiS7) on the progression of stages.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ypd3ZiUVRO", "reviewer": "Reviewer_s8oQ", "summary": "Summary:\nThe paper presents STIC, a method to enhance LVLM by reducing the need for labeled data. STIC generates image descriptions using unlabeled images and improves reasoning by reusing existing instruction-tuning data. It demonstrates performance gain across seven benchmarks, showing potential to effectively leverage vast quantities of unlabeled images for self-training.", "questions": "Have the authors changed the default latex template? There should be no anonymous submission ID. And it seems that the authors gain more space because they remove the anonymous authors part.", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Strength:\n1. The proposed method improves VLLM's performances with efficient cost during data collection. \n2. The performance improvement is consistent on diverse benchmarks and achieves an average accuracy gain of 3.8%, which is quite significant.", "weaknesses": "Weakness:\n1. Interesting idea about using different prompts to generate both good and bad captions. However, I wonder if there is any method needed to ensure the correctness of the prompts generated by the \"step-by-step\" prompt strategies. Based on my own observations, VLLM is not good at following instructions, which means the description will be even worse if the prompts are complicated. Have the authors observed similar problems?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your feedback and make clarifications as below.\n\n### **W1. If there is any method needed to ensure the correctness of the well-crafted prompt?**\n\nThank you for raising this important question. The concern regarding the complexity of prompts is indeed crucial. To address this, we implemented restrictions and human filtering on multiple candidate prompts generated by GPT-4 to ensure they function as intended. This process can be viewed as a behavior distillation from a stronger model.\n\nTo ensure effectiveness, we tested these prompts on MSCOCO samples and verified the LVLM’s instruction-following quality through human evaluation. We provide a detailed explanation along with additional experiments in our [global rebuttal (Global A1)](https://openreview.net/forum?id=FZW7Ctyjm3&noteId=VV3MK6FiS7), and we will include this discussion and the additional experiments in our revision.\n\nRegarding the concern about instruction-following abilities, which may be weaker in untrained models, we found that DPO alignment fine-tuning significantly enhances this capability. This allows the model to learn not only the preferred response but also to identify and avoid dispreferred and often erroneous responses.\n\nVerification remains an intrinsically challenging problem, especially since there is no ground truth answer in the context of image description tasks with unlabeled images. To mitigate this, we applied a human filtering strategy for the prompts, which has proven effective and not costly. To scale up our method and move towards a fully autonomous framework, we plan to involve a critic model in the process in our future studies.\n\n### **Q1. Have the authors changed the default latex template?**\n\nWe apologize for any confusion. We did not intentionally alter the template. It appears we inadvertently included the submission ID. We appreciate your understanding and will ensure this mistake is corrected. We checked that with the new template, the paper can still be fit into the page limit.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EeQIE51xCL", "reviewer": "Reviewer_uhYN", "summary": "This paper addresses the problem of acquiring high-quality fine-tuning data for large vision language models (LVLMs) with minimum human effort. The paper presents STIC (Self-Training on Image Comprehension). STIC contains two stages: image comprehension self-training and description-infused fine-tuning. LVLM first generates descriptions based on images and corrupted images and regards each sample as preferred and dis-preferred responses, respectively. The base LVLM is trained on the generated preference data leveraging the directed preference optimization (DPO) framework. Next, the LVLM is fine-tuned on instruction following data. Experimental results show that STIC outperforms the compared methods on diverse benchmarks.", "questions": "- Q1: In Algorithm 2, why aren’t the model-generated description $\\mathbf{y}_{\\mathrm{des}}$ used for fine-tuning?\n- Q2: What is the motivation of the stage 2 (Description-infused fine-tuning)? It is not directly related to the motivation of the paper (difficulty of obtaining high-quality fine-tuning data).\n- Q3: How can we validate the effectiveness of description-infused fine-tuning? Did the authors check the performance of the method which did not use the model description?\n- Q4: Table 1 just shows the performance with and without STIC. How much does each stage contribute to performance improvement?\n- Q5: Table 2 shows that the describe-and-respond (DaR) prompting method improves overall performance. DaR was not mentioned until the section for experiments. Why did not the authors describe DaR in detail in the method section?\n- Q6: Is there any reason why STIC randomly selects unlabeled images? Some self-training algorithms present the methods for selecting unlabeled data to learn.\n- Q7: How can we guarantee that all samples generated by the LVLM inputting clean images are preferred responses?\n\n**Typos**\n\nFigure 1: SPIC to STIC", "limitations": "The limitations were adequately addressed.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- S1: Overall, the manuscript is well-written and easy to read. Preliminaries and figures help readers understand the paper.\n- S2: The paper presents diverse analysis and discussion in experiments.", "weaknesses": "- W1: The manuscript has limited soundness for several reasons. For example, the authors did not validate the effectiveness of description-infused fine-tuning. In Table 2, much of the performance gain comes from the prompting method (DaR) which is outside STIC.  \n- W2: While there is a rich literature on existing self-training algorithms [1,2,3,4,5], the paper only discusses recent self-improvement systems, especially in the context of LLMs. This limited investigation leads readers to question what the technical contribution of STIC is compared with the existing line of semi-supervised learning (or self-training) research.\n- W3: The paper experimented with a fairly small amount of unlabeled images (6k and 12k images). What and how much data should be used to maximize the performance has not been thoroughly investigated.\n\n\n**References**\n\n[1] Self-training with noisy student improves imagenet classification. Xie et al., CVPR 2020.\n\n[2] Rethinking pre-training and self-training. Zoph et al., NeurIPS 2020.\n\n[3] Fixmatch: Simplifying semi-supervised learning with consistency and confidence. Sohn et al., NeurIPS 2020.\n\n[4] Revisiting self-training for neural sequence generation. He et al., ICLR 2020.\n\n[5] The dialog must go on: improving visual dialog via generative self-training. Kang et al., CVPR 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the detailed feedback. Please find our responses below. We hope that our clarifications and additional experiments resolve the misunderstanding. \n\n### **W1/Q3: In Table 2, much of the performance gain comes from DaR.**\n\nWe respectfully point out that there is a misunderstanding of the results presented in Table 2. As we have discussed in line 278-283 of our submission, Table 2 (row 1-2) showed that DaR prompting alone even results in degradations, while only combining with STIC achieves the best result. And contrary to the your conclusion, the 3rd row for **STIC without DaR** clearly showed that STIC fine-tuning itself leads to significant improvements across all tasks, with the average increasing from 54.8 to 57.6. Notably, it achieves the best results on MMBench and MM-Vet, proving its soundness. In the 4th row, we combined STIC with DaR. While DaR prompting alone is not sufficient, the integration of STIC significantly boosts the model's image comprehension ability and consequently effectiveness of DaR. \n\nMore analysis of the stages is provided in [our global rebuttal (Global A2)](https://openreview.net/forum?id=FZW7Ctyjm3&noteId=VV3MK6FiS7). \n\n### **W2: Existing self-training methods for vision models are not discussed.**\n\nThank you for mentioning the references, we will include them in discussion in our revision. Specifically: \n- **Similarity**: The major goal shared by the mentioned papers and STIC is designing an effective way of leveraging unlabeled data to improve the performance of current models. \n- **Differences**: \n    1. (Focus/Model) The mentioned papers focus on representation learning of deep learning models. Meanwhile, STIC focuses on the vision LLMs, and the backbone remains an LLM. While previous deep learning models are grounded in representation and further specialized in specific task, LLMs are autoregressive and easily generalize to different tasks. Instead of training for better image representations, STIC aims to gather synthetic data for the LLM to produce higher-quality responses to a query on an image. \n    2. (Algorithm) We focus on **alignment fine-tuning**. Notably, classic self-training algorithms for vision models do not employ alignment algorithms like RLHF or DPO. While providing a positive training signal is beneficial, having negative examples is crucial for the success of LLM alignment. As shown in Table 3 of our submission, including only positive examples for SFT is not as effective as having pairwise preference data.\n\n### **W3: A small amount of unlabeled images**\n\nIn our discussion with POVID in Section 6 Figure 5, we highlighted the **data efficiency** of STIC. While POVID uses 17k SFT data, STIC achieves better results with a total of 11k data (5k SFT and 6k unlabeled). STIC requires a smaller amount of data to achieve significant improvements. \n\nWe conducted an additional experiment using 30k unlabeled images, as shown in Figure 1 of our [attached one-page pdf](https://openreview.net/attachment?id=VV3MK6FiS7&name=pdf). The results demonstrate the scalability of STIC when applied to larger datasets. \n\n### **Q1: Why aren’t model-generated description 𝑦_des used for fine-tuning?** \n\nWe will correct this typo in Algorithm 2 in our revision. The correct data used for fine-tuning should be $([v^{(i)}, y_{des}, x^{(i)}], y^{(i)})$. Algorithm 2 indeed aims to use the model-generated description and append it before the question prompt for fine-tuning.\n\n### **Q2: Motivation of Stage 2**\n\nAs explained in our method section, stage 2 is aimed to further fine-tune the model to leverage self-generated image descriptions for downstream tasks, and thus help ground its reasoning ability on such descriptions. While stage 1 improves the model’s ability in image description, it does not focus on leveraging these descriptions for subsequent tasks such as question answering. Stage 2 fine-tunes the model by reusing a small amount of its SFT data and infusing it with self-generated descriptions to specifically strengthen model's reasoning ability based on descriptions. \n\n### **Q4: How much does each stage contribute to performance improvement?** \n\nThank you for raising this important point. Please see our response in [Global A2](https://openreview.net/forum?id=FZW7Ctyjm3&noteId=VV3MK6FiS7).\n\n### **Q5: DaR was not mentioned early.**\n\nDaR was discussed in line 211-213 of our submission and was not elaborated due to page limit. Please see our elaborated explanation in [Global A3 of our global rebuttal]((https://openreview.net/forum?id=FZW7Ctyjm3&noteId=VV3MK6FiS7)). We will add a paragraph in our method section in revision. \n\n### **Q6: Why STIC randomly selects unlabeled images.**\n\nData selection is indeed a promising future direction and we will include this discussion in revision. \n\nThe current implementation of STIC randomly selects unlabeled images as it simplifies the process and ensures a broad and diverse sampling of data. In Table 3 of our [attached one-page pdf](https://openreview.net/attachment?id=VV3MK6FiS7&name=pdf), we included an experiment using the same amount of images but more diverse distribution (Vision Flan) for stage 1. Notably, the increased diversity led to further improvements in STIC, suggesting the potential for enhancement with better sets of unlabeled images. \n\n### **Q7: How can we guarantee that all samples generated by LVLM on clean images are preferred?**\n\nRegarding the specific question on clean images vs. corrupted images, we note that this approach has become widely-used and tested in concurrent works focusing on LVLM alignment [1]. In our paper, Figures 3, 9, and 10 also showed that image corruptions indeed cause observable declines in model output quality. Please also see our response in the [global rebuttal (Global A1)](https://openreview.net/forum?id=FZW7Ctyjm3&noteId=VV3MK6FiS7) for a detailed explanation on preference alignment. \n\n[1] Aligning modalities in vision large language models via preference fine-tuning.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely thank all the reviewers for their insightful and encouraging feedback on our manuscript. We are grateful for the recognition of the significant performance gains achieved by STIC (B7oX, s8oQ), the novelty and efficiency of STIC (qFQt, s8oQ), the effective use of unlabeled images (qFQt, B7oX) and the comprehensive analysis and clarity of our manuscript (uhYN). 

In response to the comments, we provided additional experiments in the [attached one-page pdf](https://openreview.net/attachment?id=VV3MK6FiS7&name=pdf). Specifically, the results include 

1. **Scaling up to 13B model (Table 1)**: STIC is effective in improving larger-scale models as well, with further improvement potential in hyperparameter tuning, data filtering and data scaling. 

2. **Quantitative analysis of the prompt quality (Table 2)**: the prompts with better quality (our well-crafted prompt derived from GPT-4) provides better performance of DaR on the original model.

3. **Effect of enhanced image diversity in unlabeled images (Table 3**): a more diverse unlabeled image data can provide better improvement for STIC, which aligns with our ablation study on image distributions.  

4. **Effect of more unlabeled data from MSCOCO used in stage 1 (Figure 1)**: STIC scales well to larger datasets.

5. **More qualitative examples (Figures 2 and 3)** showing how STIC helped improve model performance even in mathematical VQAs. 

### **Global A1: Explanation into our prompt design and data generation for alignment fine-tuning.**

Our prompt design for the well-crafted prompt focuses on quality and diversity. We use GPT-4 to generate and sample multiple initial prompts, which are then refined through human filtering. To ensure effectiveness, we test these prompts on MSCOCO samples, verifying their ability to produce well-structured and relevant responses from the model. The restrictions we apply to reflect the quality of the prompt include length (prompt must be between 60 to 150 words to balance informativeness and conciseness), diversity (prompt includes at least 3 distinct aspects or questions about the image to encourage comprehensive analysis), and specificity (while being general, the prompt contains at least 2-3 specific cues or keywords that can be adapted to various image contents). 

More generally, instead of relying on explicit human labeling for each model generation pair as in RLHF, which can be very expensive, we adopt an "implicit" preference approach. We work under the assumption that responses generated from prompts that have differences in human preference lead to responses of the same preference with high probability. This approach allows us to create effective training data without the need for extensive human annotation. 

Our goal is thus not to identify the best or worst prompt for the task, but rather to explore the differences between them. For the design of a good prompt, we aim to guide the model to provide a comprehensive and precise image description. The bad prompts are designed to elicit inaccurate descriptions by setting up a slightly different task (describe objects that would logically exist in the image) for the model. The key is that the discrepancy between good and bad prompts should result in pairs of responses that share the same implicit preference with high probability, which is sufficient for effective DPO training.

Table 2 in our attached PDF presents additional experiments using DaR to demonstrate prompt quality. We compared normal prompts from our main paper (e.g., "Illustrate the details of the picture.") with the hallucination prompts and well-curated prompts used for DPO pair generation. The results show an expected discrepancy in QA performance: hallucination prompts significantly decreased performance, while well-curated prompts maintained a decent performance. We also included results based on a prompt proposed by Llama-3 8B and filtered using the same restrictions. The performance difference between GPT-4 and Llama-3 8B prompts underscores the quality of GPT-4's proposals. 

### **Global A2: Progression of stages.**

In the table below, we illustrate the sequential improvement in performance of STIC on ScienceQA. While stage 1 focuses exclusively on enhancing the perception capabilities of the LVLM, it still notably improves performance on downstream VQA tasks. Building on the improved image comprehension achieved in stage 1, stage 2 introduces an enhanced reasoning process that utilizes the model’s self-generated image descriptions and results in an even more significant gain. This enhancement further enables the model to self-augment its prompts with DaR, resulting in the substantial performance gains of 6.4% observed.

| Original | After Stage 1 | After Stage 2 | After Stage 2 with DaR | 
| :---: | :---: | :---: |  :---: | 
| 68.86 | 69.96 | 72.48 | 75.26 |

### **Global A3: Explanation on DaR.**

In line 211-213 and 275-284 of of our submission, we discussed DaR. Here, we provide further explanations. We proposed DaR as an additional and optional step that can be employed during inference time. Instead of directly obtaining the model's response to a particular question, DaR first prompts the model to describe the image, then appends this description to the question to finally obtain the answer:  

User: `<image>\nDetail the composition and subjects within the frame.`

Model: `<image description>`

User: `<image>\nImage description:\n<image description>\n<question>`

Mode: `<response>`

This two-step approach helps the model to better contextualize the question by grounding its response in a detailed understanding of the image. **However, as shown in Table 2 of our submission, DaR alone does not notably improve the performance of a plain model. Instead, it shows the most substantial improvement when combined with STIC fine-tuning.** The foundational improvements made by STIC on the model’s image comprehension ability consequently improved the effectiveness of DaR.

Meta-review: This paper proposes a self-training approach to enhance the image comprehension abilities of VLMs by prompting the model with good and bad prompts to generate a preference datasets. The good prompts guide the model to reason step-by-step while the bad prompts are intentionally misleading or used with corrupted images. In addition, the image descriptions of good prompts are used to augment existing instruction tuning data. Experiments show the effectiveness of STIC across many established VLM benchmarks despite using a small amount of additional unlabeled data.

Many reviewers appreciate the novelty of the STIC approach to construct preference dataset (qFQt), its data efficiency using unlabeled images (s8oQ, qFQt, B7oX) and finally the significant performance gains (B7oX, s8oQ) on common benchmarks. The reviewers are happy with the authors' rebuttal and most of them are leaning to accept the paper with overall scores (6, 6, 6, 4). The WR reviewer's main concern was about the instruction-following ability of the model when good prompts are given. The author had provided a detailed response in their rebuttal and global response A1, which has addressed the concern adequately in my opinion.

Thus, I recommend to accept this paper.
Decision: Accept (poster)