Video-STaR: Self-Training Enables Video Instruction Tuning with Any Supervision

## Abstract

Abstract The performance of Large Vision Language Models (LVLMs) is dependent on the size and quality of their training datasets.
Existing video instruction tuning datasets lack diversity as they are derived by prompting large language models with video captions to generate question-answer pairs, and are therefore mostly descriptive.
Meanwhile, many labeled video datasets with diverse labels and supervision exist - however, we find that their integration into LVLMs is non-trivial.
Herein, we present Video S elf- T raining with a ugmented R easoning (Video-STaR), the first video self-training approach. Video-STaR allows the utilization of any labeled video dataset for video instruction tuning.
In Video-STaR, an LVLM cycles between instruction generation and finetuning, which we show (I) improves general video understanding and (II) adapts LVLMs to novel downstream tasks with existing supervision.
During generation, an LVLM is prompted to propose an answer. The answers are then filtered only to those that contain the original video labels, and the LVLM is then re-trained on the generated dataset.
By only training on generated answers that contain the correct video labels, Video-STaR utilizes these existing video labels as weak supervision for video instruction tuning.
Our results demonstrate that Video-STaR-enhanced LVLMs exhibit improved performance in (I) general video QA, where TempCompass performance improved by 10 % 10\% , and (II) on downstream tasks, where Video-STaR improved Kinetics700-QA accuracy by 20 % 20\% and action quality assessment on FineDiving by 15 % 15\% .

## 1 Introduction

The advent of large Vision-Language Models (LVLMs) marked a significant milestone in artificial intelligence.
These models aim to create versatile systems capable of understanding and executing vision-and-language tasks aligned with human intentions.
Early advancements in LVLMs, as exemplified by works such as BLIP (Li et al., ; ) and LLaVA (Liu et al., ; ), have been driven by the dissemination of pre-trained large language models (LLMs) (e.g., LLaMA (Touvron et al., ; )) and pre-trained vision/vision-language models (e.g., CLIP (Radford et al., )). LVLMs connect the two model types via visual-language alignment and instruction tuning.

Figure: Figure 1: Video-STaR Overview. Video-STaR can utilize any labeled video dataset, including AR (Action Recognition), AQA (Action Quality Assessment), and TAL (Temporal Action Localization) – from which it generates video instruction tuning data (video, question, answer triplets). Internally, Video-STaR cycles between: (I) Answer Generation, where an LVLM is prompted to generate candidate answers for the questions. (II) Label Verification where generated answers are filtered to only those that contain the video labels. And (III) Instruction Tuning, where a model is retrained on answers that pass verification. These cycles continue until performance plateaus.
Refer to caption: x1.png

Wei et al. (); Liu et al. (); Karamcheti et al. () demonstrated the importance of visual instruction tuning on the resulting LVLM’s performance.
However, while much progress has been made in image-LVLMs, video-LVLMs still face challenges due to the relative difficulty in generating quality video instruction tuning datasets.
Even though videos contain more complex scene dynamics and temporal information, indicating a need for larger and more diverse training datasets compared to images, the largest video instruction dataset, VideoInstruct-$100K$ (VI-$100K$) (Maaz et al., ), comprises $100K$ video-text pairs but only $13K$ unique videos. This is small compared to image instruction datasets like LLaVA-$1.5$-Instruct (Liu et al., ), which has $\sim 665K$ image question pairs and $\sim 350K$ unique images. Such limitation in video instruction tuning leads to performance saturation (Li et al., ; Maaz et al., ; Jin et al., ; Wang et al., ; Lin et al., ).

Furthermore, due to video instruction tuning dataset construction - mainly prompting large language models to produce question-answer pairs - these video datasets often degrade to simplistic questions, prompting for video captions — $75$% of VI-$100K$’s questions are of this type (see App. Fig. 10).
Combining different sources of supervision has the potential to generate more diverse video instruction tuning datasets, enhancing video understanding.
Such supervision exists, as the broader computer vision community has developed an extensive collection of video benchmarks tailored for diverse tasks such as action recognition (Smaira et al., ; Soomro et al., ), action quality assessment (Xu et al., ; Zhang et al., ), among others (Heilbron et al., ; Wu et al., ; Grunde-McLaughlin et al., ; Grauman et al., ; Luo et al., ).

Beyond improving overall LVLM performance, adapting LVLMs to novel or out-of-domain tasks is also crucial. While LVLMs have many novel and impactful applications, many remain out of reach — such as analyzing radiology images (Senkaiahliyan et al., ), meteorological data (Lawson et al., ), traffic analysis (Zhou & Knoll, ), judging sporting events (e.g., gymnastics, Olympic diving), and assisting in surgical procedures, among others (Zheng et al., ; Jiang et al., ; Deng et al., ). These tasks require expert, in-domain knowledge that LVLMs lack, necessitating adaptation through instruction tuning.
However, collecting video instruction tuning datasets is complex and requires extensive manual effort. For instance, training an ‘AI judge’ to judge Olympic diving would traditionally involve collecting detailed expert critiques of each dive. On the other hand, these tasks often include auxiliary annotations that could be leveraged, such as surgical outcomes in medical procedures or judging scores in Olympic events.

To address these challenges, we take inspiration from LLM’s capability for self-improvement (a.k.a self-training) (Huang et al., ; Zelikman et al., ), which involves training a model on its generated data and filtering to exclude low-quality outputs.
A model’s performance is improved by cycling between generation, filtering, and training.
For example, Zelikman et al. () introduced Self-Taught Reasoners which generate chain-of-thought by prompting an LLM to generate answers and rationalize responses, retaining only correctly answered questions for further training.
Herein, we explore self-training in LVLMs and introduce Video Self-Training with augmented Reasoning (Video-STaR, see Fig. 1).
Video-STaR enables the incorporation of *any*
labeled video dataset of any format or task by prompting an LVLM with the video and a question
to generate answers (Fig. 2, 2.1) containing the video content’s label.
If the model cannot correctly answer the question, we provide the video label and ask it to rationalize it (Fig. 2, 2.2).
We then utilize the labels as weak supervision, rejecting answers that do not contain the correct label (Fig. 2, 2.3).
By facilitating the use of any supervision for video instruction tuning, Video-STaR enables the creation of diverse datasets.

Our experimental setup initializes Video-STaR with Video-LLaVA (Lin et al., ), focusing on assessing its impact on video question-answering (VQA) performance.
After a few Video-STaR training cycles, we compare the performance of Video-STaR to other state-of-the-art LVLMs and strong baselines to gauge the effectiveness of the Video-STaR framework.
Our findings demonstrate notable enhancements in accuracy and reasoning capabilities, highlighting Video-STaR’s role in overcoming the constraints posed by conventional video instruction tuning datasets.
We show that the integration of Video-STaR not only boosts Video-LLaVA’s performance on standard zero-shot VQA benchmarks but also significantly improves its adaptability to various downstream video understanding tasks. This underscores Video-STaR’s capacity to advance LVLM training while improving overall performance and versatility.

Our contributions can be summarized as follows:

- 1.
We introduce Video Self-Training with augmented Reasoning (Video-STaR), the first video self-training Large Video-Language Model method. Using self-training, Video-STaR enables the use of any labeled video dataset for video instruction tuning.
- 2.
Video-STaR improves zero-shot video question answering performance on various benchmark datasets, as evidenced by increased accuracy on TempCompass (Liu et al., ) from $45.7$ to $50.3$ ($+10\%$) and on MSVD-QA (Maaz et al., ) from $69.7$ to $71.3$ ($+2.3\%$).
- 3.
We demonstrate that Video-STaR can adapt LVLMs to diverse video tasks, notably enhancing action quality assessment on FineDiving (Xu et al., ), where accuracy rose from $17.6$ to $20.2$ ($+15\%$) and in action recognition on Kinetics700 (Smaira et al., ), where accuracy increased from $50.0$ to $59.9$ ($+20\%$).
- 4.
Utilizing Video-STaR, we create a large, $1M$ video instruction tuning dataset - VSTaR-1M, sourced from diverse datasets and tasks, and show that it benefits LVLM performance.

## 2 Video Self-Training with augmented Reasoning (Video-STaR)

Given a dataset of videos $v$ and their corresponding labels $l:\mathcal{D}=\{(v_{i},l_{i})\}_{i=1}^{d}$, Video-STaR’s objective is to create question $q$ answer $a$ pairs to instruction-tune the pre-trained model $M$ on the dataset $\hat{\mathcal{D}}=\{(v_{i},q_{i},a_{i})\}_{i=1}^{d_{f}}$, producing the instruction-tuned model $\hat{M}$. Note that videos need not be from the same task, and may contain multiple labels.
We start by prompting a large language model with a task description $T$ and video labels $L$ to generate candidate questions $q$:

$$ $\displaystyle Y_{T,L}$ $\displaystyle=\parbox[t]{348.26173pt}{A video is labeled \{L\} for the task of \{T\}. What questions could you ask someone about the video that should contain the video labels in the response? }$ $$

Video-STaR performs generation-training cycles, where in cycle $i$ the instruction-tuned model $\hat{M}^{i\star}$ is produced, while the instruction-tuned model from the previous cycle $\hat{M}^{(i-1)\star}$ is utilized for training data generation. We initialize the process with $\hat{M}^{0\star}$, an existing instruction-tuned model.

To prepare the training data in cycle $i$, answers are generated either directly via Answer Generation or through backward rationalization via Label Rationalization. In Answer Generation, $\hat{M}^{(i-1)\star}$ is prompted with questions (Sec. 2.1). Candidate answers are then filtered using the original video labels (Sec. 2.3). Videos rejected during direct Answer Generation are rationalized, where $\hat{M}^{(i-1)\star}$ is provided both a video $v_{i}$ and labels $l_{i}$, and then prompted with the question again (Sec. 2.2). Candidate answers are filtered again, creating the instruction tuning dataset in cycle $i$, $\hat{\mathcal{D}_{i}}$ of size $d_{i}$. A pre-trained model $M$ is then finetuned on $\mathcal{D}_{i}$, producing $\hat{M}^{i\star}$. The next cycle generates data using $\hat{M}^{i\star}$, until the performance plateaus (see Fig. 2).

Figure: Figure 2: Video Self-Training with augmented Reasoning. (2.1) We initialize by prompting an LVLM to generate an answer for a particular video. (2.3) We then filter the generated answers to those only containing the original video labels. (2.2) The videos whose generated answer did not contain the ground-truth labels are then sent to label rationalization, where given the video, question, and label - the model is expected to rationalize the label. (2.3) The generated answers are filtered again to those only containing the ground-truth labels, and (2) the LVLM is instruction-tuned from the pre-trained checkpoint on the resulting dataset. The cycle is then repeated.
Refer to caption: x2.png

### 2.1 Answer Generation

Each Video-STaR cycle begins in direct Answer Generation.
In this phase, $\hat{M}^{(i-1)\star}$ is prompted with the video-question pair to provide an answer along with a detailed rationale:

$$ $\displaystyle Y_{Q}$ $\displaystyle=\parbox[t]{348.26173pt}{Question: \{Q\}. Rationalize your answer step-by-step; how can one arrive at this conclusion? }$ $$

When prompted with the question $q_{i}$ on a particular video, $\hat{M}^{(i-1)\star}$ is expected to generate an answer $a_{i}$ that contains the label $\hat{l}_{i}$ and the rationale $r_{i}$ ($a_{i}=r_{i}\cup\hat{l}_{i}$, see Fig. 2).
We observe that answers containing the correct labels are of higher quality and suffer less from hallucination.
Therefore, we filter the generated answers to include only those that contain the correct label $(\hat{l}_{i}=l_{i})$ utilizing a verifier (Sec. 2.3).
For an example of Answer Generation, see Fig. 3.

### 2.2 Label Rationalization

Answer generation has two main drawbacks: (i) in some applications, especially on challenging/out-of-domain tasks, initial answer generation yield is low, resulting in almost no training samples after filtering (e.g., FineDiving, see Fig. 3); (ii) improvement plateaus as the model fails to solve new problems in the training set, and it is only trained on examples it answers correctly.

Inspired by Zelikman et al. (), for videos whose $\hat{M}^{(i-1)\star}$ generated answers did not contain the ground-truth labels, we introduce label rationalization as part of Video-STaR. Concretely, we provide $\hat{M}^{(i-1)\star}$ the video, question, and video label and instruct the model to rationalize the label:

$$ $\displaystyle Y_{Q,L}$ $\displaystyle=\parbox[t]{348.26173pt}{Question: \{Q\}. Answer: \{L\}.\\ Can you rationalize the answer step-by-step? How can one arrive at this conclusion? }$ $$

Given the correct label, the model can reason backward and more easily generate a rationale leading to the correct answer. However, label rationalization is more prone to hallucinations, so we prefer direct answer generation and use rationalizations only if answer generation fails. For an example of Label Rationalization, see Fig. 3, top. The generated answers are then filtered again, keeping only those that contain the correct label $(\hat{l}_{i}=l_{i})$ utilizing a verifier (Sec. 2.3).
Label Rationalization is only utilized in training cycles; only direct answers are used to produce the final model $\hat{M}^{\star}$.

### 2.3 Label Verification

Video-STaR aims to utilize the labels as weak supervision in instruction tuning data generation. Gold labels are a grounding aspect of our datasets and represent some ground-truth knowledge. Therefore, we assume that answers that contain the ground-truth labels in their responses are higher quality than those that don’t.
While we would like to validate the existence of the different labels in the generated text, this can be non-trivial.

To this end, we introduce the Parser-Verifier. The Parser, $P$ extracts the predicted labels from the generated text ($\hat{l_{i}}=P(a_{i})$), using a mixture of named entity recognition and Regex. Regex is used to identify easily identifiable string patterns, such as bounding boxes and time ranges, while named entity recognition is used for more nuanced entities, such as timestamps.
The Verfier, $V$ compares the extracted labels with the gold ones using the appropriate metrics ($V(l_{i},\hat{l_{i}})\rightarrow\mathbb{R}$). For example, IoU for bounding boxes/temporal action localization, and BERT (Devlin et al., ) embedding similarity for sentence ordering. Each video has between 1-3 associated labels. To be classified as correct, the predicted labels must be within a 5% margin of error from the gold.

Table: Table 1: Source Dataset Summary. Video datasets used as source datasets for instruction tuning data generation, their source tasks, and the number of unique labels.

## 3 Video-STaR Generated Dataset - VSTaR-1M

In this section, we detail the different source datasets utilized in our study (Sec. 3.1) and analyze the generated Video-STaR Dataset, VSTaR-1M (Sec. 3.2).

Figure: Figure 3: Qualitative Improvement of Data Generation over Cycles on FineDiving. We initialize the model with Video-LLaVA (Cycle 0), where the model cannot generate an answer ($\rightarrow|\times$) or rationalize the label correctly ($|\rightarrow\times$). In the second cycle (Cycle 1), the model still cannot generate an answer ($\rightarrow|\times$) but can rationalize the video label ($\checkmark|\rightarrow$), which is selected for instruction tuning. Finally, in the third cycle (Cycle 2), the model directly generates a correct answer ($\checkmark|\rightarrow$), which is selected for visual instruction tuning. We highlight in green correct answers, in red wrong answers, and in yellow - hallucinations.
Refer to caption: x3.png

### 3.1 Source Datasets

In selecting source datasets, we selected datasets that contain diverse video content and label types, please see Tab. 1.
These include Kinetics700 (Smaira et al., ), which has action recognition annotations and is particularly large and diverse. FineDiving (Xu et al., ) is an action quality assessment dataset of Olympic diving events and has both an overall score and action sequence annotations. Finally, STAR-benchmark (Wu et al., ), a video reasoning dataset,
also contains bounding box and temporal action localization annotations. Tab. 1 contains the relevant dataset statistics, e.g., the number of videos and labels per dataset.

### 3.2 Generated Dataset Analysis

#### Quantitative Analysis.

Through the application of Video-STaR, significant dataset augmentation was achieved over two cycles of, illustrated in Fig. 4.
This figure displays the Answer Generation and Label Rationalization yield across the datasets source.
Notably, the initial application of Video-LLaVA on datasets like Kinetics700 and STAR-Benchmark showed significant Answer Generation success rates.
However, the FineDiving dataset presented a notable challenge, with Answer Generation having no answers generated directly, underscoring the complexity of the dataset and the critical role of Label Rationalization. By the end of the second cycle, a substantial number of high-quality instances had been produced, showcasing both the effectiveness of Video-STaR in converting labeled video datasets into video instruction tuning dataset, as evidenced in Fig. 4.

#### Qualitative Analysis.

See Tab. 2 for examples of generated question-answer pairs.
From Kinetics700, we extracted an instance showcasing a video labeled ‘smashing’. Video-STaR correctly identified a more fine-grained label, ‘chopping wood’.
In the FineDiving dataset, a clip depicting a complex dive was accompanied by the question ‘On a scale from 1-100$\cdots$’ The model’s output text provided a breakdown of the dive’s components, leading to a score (label), as would be desired from an LVLM visual assistant.
Finally, in the STAR benchmark, questions are already provided; therefore, we utilized them directly.

**Table 2: Examples of Generated Data. Examples of the video, question, Video-STaR generated answer, and ground-truth label from each source dataset.**
|  | Kinetics700 | STAR-Benchmark | FineDiving |
| --- | --- | --- | --- |
| Video |  |  |  |
| Question | What type of activity is being performed in this video? | Which object was put down by the person? | On a scale from 1-100, how would you evaluate the execution of the actions in the video? |
| Generated<br>Answer | In the video, a man is seen holding a large axe and swinging it around while a crowd watches. He then proceeds to hit a block of wood with the axe. Based on these observations, it can be concluded that the activity is chopping wood. | In the video, the person is seen picking up a bag and putting it down on the floor. After that, he picks up a book and puts it down on the floor. Based on this information, we can conclude that the person put down the bag and picked up the book. | First, we identify the diver’s action sequence: inward somersault $\cdots$ entry into the water. Evaluating each movement’s difficulty, we assign $\cdots$ inward somersault: 2.5, $\cdots$. By averaging these scores $\cdots$ the overall score is 63.0 out of 100. |
| Label | Smashing | The bag. | Overall Score 63.0 |

Figure: Figure 4: Dataset Yield vs. Cycles. Percentage of the videos converted to instruction tuning by the Answer Generation and Label Rationalization per dataset. As can be seen, on difficult datasets, such as FineDiving, no videos are converted by Answer Generation in the first cycle. By utilizing Label Rationalization, the model is able to improve to eventually generate answers correctly.
Refer to caption: x4.png

In 3, we show the qualitative improvement of the generated data over Video-STaR cycles.
In the first cycle (Cycle 0), Video-LLaVA failed at Answer Generation and Label Rationalization. After one Video-STaR cycle (Cycle 1), Video-STaR still failed at Answer Generation but succeeded in Label Rationalization. After the final Video-STaR cycle (Cycle 2), Video-STaR managed to generate the answer without requiring the label via Answer Generation.

## 4 Experiments

We experimented with Video-STaR and evaluated its enhanced video understanding capabilities. In Sec. 4.3, we evaluate how Video-STaR adapts Large Vision-Language Models (LVLMs) to the source datasets and how these capabilities are transferred zero-shot to similar benchmarks. In Sec. 4.2, we evaluate the video question-answering capabilities on video benchmark datasets.

### 4.1 Experimental Setting

#### Implementation Details.

We initialize from the Video-LLaVA (Lin et al., ) model, which utilizes the Vicuna-7B v1.5 (Chiang et al., ). We ran three Video-STaR cycles, and each cycle was initialized with the pre-trained Video-LLaVA weights.
We train for one epoch using a $128$ batch size, AdamW optimizer, and a cosine learning rate schedule. The learning rate is $2e-5$ with a $0.03$ warmup ratio.
In combination with the generated Video-STaR instruction tuning dataset, we additionally utilized the VideoInstruct-$100K$ (Maaz et al., ) and the LLaVA v$1.5$ instruction tuning datasets (Liu et al., ).
Additional details are available in the appendix.

#### Baselines.

Besides comparing to Video-LLaVA, we also wanted to evaluate the effect of utilizing additional data and naively adapting the source datasets. Therefore, we utilized simple templates to generate question-answer pairs from the video labels and trained Video-LLaVA on the resulting dataset. We will reference this baseline as Video-LLaVA^+.
Another baseline for adapting Large Vision-Language Models to novel tasks is model distillation, where a stronger video model - in this work, Gemini $1.5$ pro-vision - is utilized to label/annotate a small set of videos ($500$ from each dataset) and used to finetune the models. We will reference this baseline as Vid-LLaVA^Gemini.

#### Evaluation Details.

We evaluate on the following benchmarks; the Zero-shot question-answer (QA) benchmarks: MSVD-QA, MSRVTT-QA, TGIF-QA, and ActivityNet-QA (Xu et al., ; ; Jang et al., ; Heilbron et al., ). TempCompass (Liu et al., ), a multiple-choice fine-grained QA benchmark.
Adapted task performance is evaluated by converting source datasets using simple templates and applying the same evaluation protocol as Maaz et al. (Maaz et al., ), producing Kinetics700-QA, STAR-benchmark-QA, and FineDiving-QA.
This protocol reports two metrics: accuracy (the percentage of correctly answered questions) and the average score (where ChatGPT rates each response on a scale of 1-5 and calculates the mean of these scores).
All evaluations utilize the same GPT model (Wu, ) (“gpt-3.5-turbo”) to ensure consistent comparisons.
Due to cost considerations, 1000 videos were randomly selected from each dataset for Gemini evaluation. The reported values are used on ActivitlyNet-QA.

**Table 3: Comparison with state-of-the-art methods on TempCompass. TempCompass (Liu et al., ) assesses the temporal understanding capabilities of video language models across five dimensions Video-STaR improves Video-LLaVA performance on TempCompass by $10\%$.**
|  | Action | Direction | Speed | Event | Attribute Change | Avg. |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | Fine | Coarse | Obj. | Cam. | Abs. | Rel. | Order | Color | Size | Both | Other |  |
| Random | 39.7 | 40.1 | 39.8 | 39.0 | 40.8 | 42.0 | 41.5 | 40.4 | 39.9 | 38.9 | 39.4 | 40.5 |
| mPLUG-Owl () | 48.8 | 66.1 | 38.7 | 36.8 | 42.2 | 38.4 | 42.0 | 41.7 | 44.7 | 41.9 | 39.9 | 44.4 |
| Video-LLaVA () | 63.4 | 93.5 | 36.1 | 34.8 | 42.7 | 26.5 | 39.1 | 52.6 | 37.1 | 43.3 | 33.3 | 45.7 |
| Video-LLaVA^+ | 62.1 | 93.0 | 35.0 | 32.6 | 41.1 | 38.7 | 36.4 | 59.0 | 40.2 | 36.7 | 44.4 | 47.2 |
| Vid-LLaVA^Gemini | 30.7 | 30.1 | 37.8 | 40.0 | 41.8 | 42.4 | 21.5 | 50.4 | 49.9 | 38.0 | 37.4 | 38.2 |
| Video-STaR | 68.6 | 94.1 | 35.8 | 38.0 | 38.7 | 37.6 | 37.1 | 53.8 | 48.5 | 45.0 | 55.6 | 50.3$(+10\%)$ |
| Gemini-1.5 () | 94.8 | 98.4 | 43.6 | 42.4 | 65.3 | 48.7 | 55.6 | 79.5 | 59.8 | 70.0 | 66.7 | 66.0 |

Table: Table 4: Zero-shot Video QA benchmarks. As can be seen, many models are approaching Gemini performance - indicating that LVLMs may be operating near the noise level on these benchmarks.

### 4.2 Quantitative Evaluation on Zero-Shot Benchmarks

To evaluate Video-STaR’s effect on general video question answering, we evaluated its effect on Video-LLaVA’s performance on TempCompass, see Tab. 3.
On TempCompass, Video-STaR outperformed Video-LLaVA across the board– by $\sim 10\%$.
To see if this performance boost is simply a factor of training on a larger dataset, we also evaluated Video-LLaVA^+. Video-LLaVA^+was trained on even a larger video dataset by naively utilizing video labels, and yields a more modest improvement of $3\%$, showing the utility of Video-STaR.
TempCompass is also a fine-grained dataset that would be sensitive to hallucinations, indicating that Video-STaR is not more prone to hallucinations compared to existing methods. Gemini 1.5 pro scored an impressive $66.0$ on TempCompass, showing there is still much room for improvement on this benchmark.

We then continued and evaluated Video-STaR’s effect on zero-shot video QA performance on the MSVD-QA, MSRVTT-QA, TGIF-QA and ActivityNet-QA benchmarks. As can be seen in Tab. 4, Video-STaR achieves performance improvements where, for instance, on the MSVD-QA dataset, Video-STaR attains the highest accuracy of $71.3$% vs Video-LLaVA’s 69.7.
On MSRVTT-QA, Video-STaR leads with an accuracy of $58.2$% and maintains a competitive edge in other datasets like TGIF-QA and ActivityNet-QA.
Seeing the relatively small performance gains compared to TempCompass, we additionally evaluated Gemini $1.5$ pro-vision on $1000$ video subsets of each dataset and found that its performance is on par with existing open-source models.
We believe this shows that we are near the ‘noise’ limit of these benchmarks.
Our qualitative analysis indicated that many of the questions selected as ‘wrong’ are actually due to the benchmark design—overly general questions with multiple correct answers. Concurrent work (Wu, ) has similarly concluded that the ChatGPT-$3.5$ version utilized in evaluation can lead to variations of $\pm 10$ in accuracy.

Table: Table 5: Adapted Dataset Performance. Performance metrics on test sets of Kinetics700, FineDiving, and STAR-benchmark datasets via converting them to QA following Maaz et al. (). Video-STaR shows significant improvement over Video-LLaVA and Video-LLaVA^+, showing the potential of Video-STaR for LVLM adaptation to new tasks.

### 4.3 Quantitative Evaluation on Adapted Datasets

Besides improving general visual question-answering performance, Video-STaR can also adapt Large Vision-Language models to novel takes. To demonstrate this, we converted the test sets (not included in training) of the source datasets – Kinetics700, STAR-benchmark, and FineDiving. The results of these evaluations are reported in Tab. 5.
Adapting LVLMs with easier-to-collect labels can be helpful in various applications, leading to a more versatile, multi-domain capable assistant.
When evaluating Video-STaR’s impact on LVLM performance on the diverse source datasets, we found that it significantly improves model performance, particularly on complex tasks. For instance, on Kinetics700, known for its extensive action categories, Video-STaR enhanced Video-LLaVA’s performance accuracy by an average of 20% (as can be seen in Tab. 5), showcasing its ability to develop generalized models adept across multiple domains. Interestingly, Video-LLaVA^+’s performance did not improve compared to Video-LLaVA, and in some cases, even worsened, showing that one cannot directly utilize labeled datasets for LVLM adaptation.

Action Quality Assessment (AQA) is a complex video task requiring detailed action understanding, where Video-STaR significantly enhanced LVLM performance on the FineDiving dataset. Our results show a notable improvement from $17.6$ to $20.2$ in score prediction accuracy, highlighting Video-STaR’s effectiveness in refining LVLM’s temporal reasoning.
However, Video-STaR allows LVLMs to not only rate a particular dive but also explain the rationale behind each assessment. This rationale is invaluable for many applications, effectively providing potential user feedback for improvement.
This advancement enables novel applications, from sports coaching to automated feedback systems, by offering evaluations and constructive feedback.
The ability to interpret and improve action quality underscores the potential of Video-STaR, underscoring the potential of utilizing LVLMs as intelligent and informative visual assistants. For more, please see App. Sec. A.2.

Table: Table 6: Ablations on Adapted Datasets. Performance metrics on test sets of Kinetics700, STAR-benchmark, and FineDiving datasets. Label Rationalization impacts mostly the difficult datasets, such as FineDiving, whose initial Answer Generation yields are low.

Table: Table 7: Ablations on Zero-Shot Benchmarks. In simpler benchmarks, Answer Generation proved more critical for zero-shot generalization than Label Rationalization.

### 4.4 Ablations

In our ablation studies, we evaluated the impact of removing Label Rationalization and Answer Generation from Video-STaR, focusing on adapted datasets (Kinetics700, FineDiving, STAR-benchmark) and zero-shot benchmarks (MSVD-QA, MSRVTT-QA, TGIF-QA, ActivityNet-QA).

#### Adapted Datasets

For adapted datasets (Tab. 6), excluding Label Rationalization led to a significant performance drop in FineDiving, from $20.2$ to $12.8$ in accuracy, highlighting its critical role in complex reasoning tasks. This is likely due to the lack of conversion of any examples from the data.
However, the removal of Answer Generation resulted in a more pronounced and uniform decline across all datasets. For example, Kinetics700’s accuracy was reduced from $59.9$ to $50.0$, underscoring its foundational role in generating context-relevant responses.

#### Zero-shot benchmarks

In zero-shot benchmarks (Tab. 7), the removal of Label Rationalization had a mixed impact, slightly affecting MSVD-QA where accuracy decreased from $71.3$ to $70.6$. The elimination of Answer Generation consistently lowered performance, such as a decrease in MSRVTT-QA accuracy from $58.2$ to $57.4$.
ActivityNet-QA performance improved, probably because $100K$-Instruct utilizes ActivityNet for instruction tuning. Therefore, the introduction of additional videos decreases performance.

## 5 Related Works

### 5.1 Large Vision-Language Models

Initial LVLMs, such as LLaVA (Liu et al., ; ) and BLIP-2 (Li et al., ), demonstrated the potential of merging image inputs with large language models. Methods like mPLUG-Owl (Ye et al., ) and Flamingo (Alayrac et al., ) further allowed for multiple image inputs without architectural changes. Li et al. () and Zhang et al. () led the transition to video understanding, integrating video/image encoders and LLMs while training on small video instruction tuning datasets.
Jin et al. () introduced Chat-UniVi, a unified model employing dynamic visual tokens for both images and videos, optimizing visual token usage and higher frame count sampling.
LLaMA-VID (Li et al., ) showed that the token count can be further reduced by pooling the tokens selectively via the text prompt using Q-Former.
Recently, Video-LLaVA (Lin et al., ) used modality-specific encoders for video and image inputs to leverage LanguageBind encoders as they are constructively aligned during pretraining and utilized a shared projection.

Maaz et al. () expanded the field with the first large video instruction tuning dataset, VideoInstruct-$100K$.
VideoInstruct-$100K$ was generated from ActivityNet (Heilbron et al., ) by prompting chatGPT with the video captions, generating question-answer pairs.
While driving much of the performance improvement in the field (Jin et al., ; Wang et al., ; Lin et al., ; Li et al., ), upon examination of VideoInstruct-$100K$, it is evident that it suffers from quality issues.
The questions often degrade into de facto prompts for a video caption (see Fig. 10) and rarely require many spatiotemporal capabilities, which may limit LVLM performance.

### 5.2 Large Language Models and Self-Training

The advent of GPT (Radford et al., ; Brown et al., ) marked significant milestones in natural language processing, showcasing LLMs’ power in understanding and generating human-like text. Open-source LLMs like LLaMA (Touvron et al., ; ) and their instruction-tuned variants like Alpaca and Vicuna (Taori et al., ; Chiang et al., ) further tailored these models for nuanced human-AI interactions.
However, even LLMs have found it challenging to scale annotated datasets for training, prompting work on self-training and self-improvement (Singh et al., ; Huang et al., ; Ho et al., ; Marasović et al., ; Hosseini et al., ). In this line of work, LLMs cycle between instruction-tuning data generation and instruction tuning, iteratively improving LLM performance over cycles. For instance, Zelikman et al. () introduced the
the Self-Taught Reasoners method, used rationalization to generate chain-of-thought (CoT) reasoning, filtering poor rationalizations to retain correctly answered questions. Other self-training approaches include expectation-maximization-based approaches (Singh et al., ), which alternate between data generation and improvement between training cycles. Alternatively, majority voting has also been utilized to generate answers and rationale for unlabeled questions (Huang et al., ). These methods show the effectiveness of iterative self-training. In our work, we aim to introduce a weakly supervised self-training approach for video instruction tuning, leveraging video supervision that is often easier to collect and exists in many large and diverse datasets.

## 6 Conclusions

In conclusion, Video Self-Taught Reasoners (Video-STaR) presents a novel approach to enhance Large Vision-Language Models (LVLMs) by enabling the use of diverse labeled video datasets for visual instruction tuning. This method addresses critical data diversity and quality challenges, leading to significant performance improvements across various video understanding tasks. Our experiments demonstrate Video-STaR’s effectiveness in both source dataset adaptation and zero-shot generalization, showcasing its potential in advancing LVLM capabilities for complex video content.

The promising results of Video-STaR open new research avenues, particularly in expanding LVLM knowledge bases using readily available image and video datasets. Future work could explore advanced self-training techniques and integration with emerging LVLM architectures, focusing on long-form video understanding to further boost LVLM understanding.
Additional work is also needed to reduce hallucinations, perhaps by using grounded VLMs as auxiliary input.

Acknowledgements. We thank Google Research for providing financial support through a Stanford HAI–Google collaboration. This work was partially supported by the National Science Foundation under Grant No. 2026498. We thank the Knight-Hennessy Scholars Foundation for funding OZ.

## 7 Limitations

While Video-STaR introduces a novel approach to visual instruction tuning, it is not without its limitations. Firstly, the methodology can be computationally intensive due to the cycling of both generating and rationalizing question-answer and instruction tuning. Secondly, the assumption that all labels necessitate a rationale may not always hold true. Certain labels might be straightforward enough not to require elaborate rationalization, potentially leading to unnecessary computational overhead. Lastly, hallucinations, especially in label rationalization can be further reduced by perhaps implementing additional verifiers.

## Appendix A Appendix

In Sec. A.1, we provide additional implementation details and compute used in developing Video-STaR.
In Sec. A.2, we introduce explainable action quality assessment and provide good and bad examples of Video-STaR on the FineDiving test dataset.
Finally, we provide additional qualitative Answer Generation and Label Rationalization examples in Sec. A.3 and A.4.

### A.1 Implementation Details

Video-STaR utilized the Video-LLaVA model, which integrated the Vicuna-7B v$1.5$ for language processing and ViT-L/14 video and image encoders from LanguageBind for visual encoding. The system’s tokenizer, adapted from LLaMA, has a vocabulary size of around $32,000$ classes and a dimensionality of $4096$.
Two cycles of Video-STaR were executed, each initialed with the pre-trained Video-LLaVA model (before instruction tuning). The training data was augmented by incorporating VideoInstruct-$100$K and LLaVA v$1.5$’s visual instruction datasets.

Four clusters of $10$ NVIDIA Titan RTX GPUs were employed for $64$ hours. The structured prompts for these tasks were as follows:

- •
Answer Generation
Question: {Q}.
Can you explain step-by-step how one can arrive at this
conclusion?
- •
Label Rationalization
Question: {Q}
Answer: {L}.
Can you explain step-by-step how one can arrive at this
conclusion?

Figure: Figure 5: Action Quality Assessments by Video-STaR on the FineDiving Test Set. Different diving sequences with corresponding Video-STaR evaluations, from a high score of $85.78$ for complex sequences (top) to $74.8$ for intermediate (middle), and a lower score of $54.6$ for basic sequences (bottom), showcasing Video-STaR’s proficiency in scoring dives with varying degrees of difficulty and execution quality.
Refer to caption: x5.png

These prompts guided the model in producing detailed answers and rationalizations, enhancing the depth and utility of the generated instruction-tuning dataset.
Answer correctness was evaluated using template matching with Levenshtein Distance-based Levenshtein () fuzzy logic Cohen (), considering an answer correct if all keywords from the label were present in the generated response with a minimum similarity score of $80$%. For example, if in Kinetics the action label is ‘eating apple pie’, we would only consider a generated answer correct of ‘eating’, ‘apple’, ‘pie’ all appeared with a similarity score of $80$.

### A.2 Explainable Action Quality Assessment

Explainable Action Quality Assessment (AQA) is critical for detailed analysis of performances in precision sports, such as competitive diving, where execution and complexity significantly impact scores. Unlike previous AQA works, which only provide a score Xu et al. (; ); Yu et al. (); Zhang et al. () Video-STaR not only generates scores but also offers detailed justifications akin to expert analysis.

Fig. 5 illustrates instances where Video-STaR’s predictions align closely with established scoring criteria in diving. For example, the model breaks down a dive sequence, Reverse$\rightarrow$$3.5$ Soms.Tuck$\rightarrow$Entry, with an assigned difficulty of $3.5$, into its components. It then logically assigns scores to each element, such as $1$ for more straightforward maneuvers and $4$ for more complex twists, culminating in an overall difficulty score of $3.3$, close to the GT $3.5$. The final predicted score was $79.2$, close to the GT score of $85.78$.

Figure: Figure 6: Challenges in Action Quality Assessment by Video-STaR on the FineDiving Test Set. Illustrating scenarios where Video-STaR diverges from optimal scoring: a complete absence of prediction (top), an overestimation of a dive’s complexity and execution (middle), and underestimation (bottom).
Refer to caption: x6.png

Figure: Figure 7: Answer Generation Across Datasets by Video-STaR. Interpreting actions on Kinetics700 (top), detailing action sequences in STAR-benchmark (middle), and evaluating diving action quality in FineDiving (bottom).
Refer to caption: x7.png

Video-STaR’s proficiency extends to dives with varying levels of performance. It can discern relatively complex dives with ground-truth scores of $74.8$ and $54.6$, which Video-STaR scored as $76.5$ and $47.0$, respectively. In both cases, the model breaks down the actions into sub-actions and rates them in terms of difficulty and execution. While it manages to rate the dives themselves well, in one instance, the model erroneously calculated the average for $2.5,3.4,3.4,3.4$ as $3.4$.

Challenges in maintaining consistent AQA accuracy are depicted in Fig. 6, showcasing instances of either the model not following instructions or estimating the score incorrectly. In Fig. 6, top, the model did not produce a score and resorted to explaining how a score might be derived.
The model occasionally produces an inaccurate score, and either grossly over (Fig. 6, middle) or under (Fig. 6, bottom) estimates the score. For example, it attributed a score of $72.0$ to a sequence deemed to have a lower difficulty level of $35.0$. These discrepancies underscore the necessity for ongoing improvements in the model’s grasp of nuanced scoring criteria to enhance reliability in AQA predictions.

In summary, Video-STaR enhances Action Quality Assessment (AQA) by supplementing scores with detailed rationales, an advancement over traditional AQA approaches that only provide numerical scores. Although it effectively dissects the components of diving performances, indicating both complexity and execution as seen in Fig. 5, it faces challenges in maintaining consistent accuracy, particularly in aligning scores with established benchmarks, as evidenced in Fig. 6. These instances highlight the need for a deeper understanding of each movement’s difficulty and execution quality to ensure the model’s scoring aligns with professional judging standards. Additionally, they emphasize the ongoing requirement to improve the model’s accuracy and ground its rationalizations in the verifiable aspects of the performances it evaluates.

### A.3 Qualitative Analysis of Answer Generation

Fig. 7 presents our model’s capabilities in generating answers and rationalizing actions across varied video contexts, demonstrating its adeptness in interpreting complex scenes.

#### Kinetics700.

In the first example from Kinetics700 (Fig. 7, Kinetics700, top), the model effectively identifies a man’s act of pretending to be a statue and further discerns the performance’s subtle aspects, such as the engagement level and the humor conveyed through a sign.
In another Kinetics700 example (Fig. 7, Kinetics700, bottom), the model processes a scene with multiple concurrent actions. Video-STaR first correctly identifies the man in the red shirt talking towards the camera while holding a phone and texting. It correctly identifies the next scene, where another man is texting while driving.
This precision in temporally locating different actions in the videos is invaluable for visual instruction tuning and could potentially enhance models’
The capability to analyze scenes with multiple focal points is an essential feature for comprehensive video understanding.

Figure: Figure 8: Successful Label Rationalizations by Video-STaR. The model accurately infers a preparatory behavior from a person’s interaction with household items (top), correctly interprets box-handling actions (middle), and provides a nuanced breakdown of a complex diving sequence, assigning an overall score of $71.4$ (bottom), exemplifying precise action understanding on FineDiving and STAR-benchmark.
Refer to caption: x8.png

Figure: Figure 9: Challenges in Label Rationalization by Video-STaR. Instances of rationalization errors are shown: an incorrect inference about dishwashing (top) and fabricated details about closet interactions (middle). A rationalization for a diving sequence (bottom) is accurate but demonstrates the model’s vulnerability to hallucinations in complex action sequences within FineDiving and STAR-benchmark datasets.
Refer to caption: x9.png

#### STAR-benchmark.

In the first STAR-benchmark example (Fig. 7, STAR- benchmark, top), the model uses inferential reasoning to deduce the purpose behind a person consuming medicine despite the absence of explicit details about the medicine. This instance showcases the model’s ability to apply logical assumptions to fill in informational gaps, a valuable trait for interpreting actions without fully detailed context. In the next example (Fig. 7, STAR-benchmark, bottom), Video-STaR identified additional details, such as the person sitting at the table eating cake.

#### FineDiving.

in Fig. 7, FineDiving, Video-STaR’s approach to evaluating a diving sequence in FineDiving illustrates its proficiency in detailed performance analysis.
By deconstructing the dive into individual elements and assessing each for difficulty, the model mirrors the evaluative processes of human judges, providing a comprehensive performance score.
This depth of analysis, which includes a critique of the dive’s complexity and execution, underscores the model’s utility in contexts requiring nuanced assessment, such as athletic performance evaluation.

These examples from Fig. 7 show how the proposed Answer Generation is capable of creating valuable and informative video question-answer pairs that can be utilized in instruction tuning, highlighting its potential applicability in various domains that demand a deep understanding of video scenes.

Figure: Figure 10: VideoInstruct-$100K$ Qualitative Evaluation. Evaluation of VideoInstruct-$100K$’s question distribution. A wide gap can be seen between ‘grounding’ questions, which contains ‘where/when’ (bottom) and the top-50 most common questions (top). when analyzing the top-50 most common questions (top), it is clear they are all prompting for video captions.
Refer to caption: figures/shared_questions_sns.png

### A.4 Qualitative Examples of Label Rationalizations

Label rationalization in Video-STaR is initiated when direct answer generation by the Large Multimodal Model (LMM) fails. Although this method proves beneficial in certain situations, it occasionally leads to the generation of hallucinated content—incorrect details or inferences not supported by the video.

#### STAR-benchmark.

Illustrated in Fig. 8, effective label rationalizations provide added context, enriching the model’s responses. A notable example from the STAR benchmark demonstrates the model’s capacity to build upon a basic action, like lying on a couch, by inferring additional activities, such as donning a shirt and tidying up, hinting at the individual’s subsequent actions. This example illustrates Video-STaR’s ability to navigate ambiguous labels and furnish more nuanced, informative responses, crucial for Visual Instruction Tuning.

However, Label Rationalization is also more prone to hallucinations. Fig. 9 exposes the model’s tendency for hallucinations, mainly where it introduces actions and details not evidenced in the video. In Fig. 9, STAR-benchmark, top; the LMM hallucinated that after taking the dish, the man washed it and placed it on a dry rack, which did not occur in the video. In Fig. 9, STAR-benchmark, bottom; the model hallucinated that the camera panned to the right, that it could see the reflection of the man in the mirror, and that he took out shoes from the closet — none of which occur in the video.

#### FineDiving.

Label rationalization proved especially useful in challenging, domain-expert datasets such as the Olympic diving scoring dataset - FineDiving.
Answer Generation initially had zero yield, and Label Rationalization allowed the model to start learning about this challenging task.
For example, in Olympic diving events, to get an overall score for the dive, one removes the top and bottom $2$ scores (out of a total of $7$), then multiplies this score with the dive difficulty. In Fig. 8, FineDiving, the LMM correctly deduced that the execution score is multiplied by the difficulty.
With no supervision, the model correctly deduced the rules, deducing that the final execution score is obtained by multiplying the execution score and the action difficulty.

Fig. 9, FineDiving, Label Rationalization failed to generate an answer with sufficient depth. Rather than providing additional insights, the model resorted to re-iterating the labels.

In summary, Label Rationalization produces shorter, less informative answers than Answer Generation and is more prone to hallucinations. It is primarily utilized so complex datasets, such as FineDiving, can be learned, especially in cases with initial zero yield.

### A.5 Evaluation of Video Instruction Tuning Datasets

We performed a qualitative evaluation of VideoInstruct-$100K$ and found that the broad majority of the questions essentially prompt the Large Vision-Language model for video captions - see Fig. 10. As can be seen, $\sim 75\%$ of VideoInstruct-$100K$’s questions are of this type.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 6
presentation: 2
questions: 1. **Clarification on Dataset Labeling**: Can the authors clarify how they handle datasets where video labels are ambiguous or not well-defined? It would be helpful to understand how such cases are managed during the label verification step to avoid generating incorrect instructions.

2. **Self-Training Cycles**: Could the authors provide more details on how they determine when the self-training process has plateaued? Is there a specific performance threshold or metric used to stop the cycles?

3. **Hallucination Mitigation**: The paper acknowledges hallucination as a potential issue, especially in label rationalization. Are there additional methods or heuristics that the authors could explore to reduce hallucinations during both answer generation and rationalization phases?

4. **Generalizability to Complex Tasks**: Video-STaR shows notable improvements in performance on diverse video tasks. Could the authors elaborate on how the method handles particularly complex reasoning tasks beyond action recognition and quality assessment? How might this approach generalize to more abstract tasks, such as subjective video analysis?

5. **Computational Overhead**: Given the iterative nature of the self-training process, what are the computational requirements and trade-offs for implementing Video-STaR at scale? Would this method be feasible for larger video datasets, and if so, what optimizations could be applied?
soundness: 2
strengths: - **Originality**: Video-STaR introduces an innovative self-training approach for video instruction tuning, enabling the use of any labeled video dataset for instruction tuning. This represents a novel solution to a critical problem in video QA and reasoning by removing the dependency on manual dataset collection.

- **Quality**: The paper is well-structured, with detailed experimental evaluations demonstrating significant improvements in video QA accuracy and adaptability to downstream tasks. It also addresses weaknesses in previous methods, such as reducing hallucinations in model answers.

- **Clarity**: The methodology, including the self-training cycles of generation, filtering, and tuning, is clearly articulated, with effective visual aids to illustrate the process.

- **Significance**: The paper contributes meaningfully to the field of video QA and LMM adaptation, showing improvements in both general video understanding and task-specific performance. It also creates a large and diverse dataset (VSTAR-1M), further enriching the resources available for multimodal research.
summary: The paper introduces a novel self-training approach, Video Self-Training with augmented Reasoning (Video-STaR), to improve the performance of Large Multi-modal Models (LMMs) in video instruction tuning. By leveraging any labeled video dataset, Video-STaR enhances video question answering and adapts LMMs to new downstream tasks. The method cycles between generating answers, verifying them using the video labels, and fine-tuning the model. Experimental results show significant improvements in video understanding tasks, such as a 6.1% increase in Video QA performance and enhanced action quality assessment accuracy.
weaknesses: - **Computational Intensity**: The iterative nature of the self-training cycles, involving both answer generation and label rationalization, introduces significant computational overhead. This could limit the scalability and efficiency of the proposed method, especially in resource-constrained environments.

- **Hallucination in Rationalization**: The reliance on label rationalization, especially for difficult tasks, may increase the likelihood of hallucinations. This undermines the robustness of the system in generating accurate explanations and answers, particularly in complex video tasks like FineDiving.

- **Generalizability of Label Rationalization**: While the method assumes that all video labels require rationalization, certain straightforward labels may not benefit from this additional step. This leads to unnecessary computational load without proportional gains in performance.

- **Limited Dataset Variety**: The study largely focuses on specific datasets like FineDiving, STAR-benchmark, and Kinetics700. Expanding the evaluation to a broader set of tasks could help demonstrate the full generalizability and adaptability of the approach to more diverse real-world video datasets.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: * The VSTAR-1M has about 1M videos mentioned in the paper but the experiments only tuned with 550k videos shown in Table 4. May I ask the authors if we apply the full VSTAR videos to tune LMMs, whether the results would be better?
* Since neither the paper nor the appendix describes the verify process carefully, I briefly skimmed through the code and found that there is a 'verifier_type' for each sample. Is the type manually labeled for different samples or is it automatically judged by some rules?
* In the paper it is mentioned that bounding boxes can be used as labeled data, so does this data enable LMMs to localize objects better than before? For example, to get better results when doing video segmentation or object tracing or simply image object detection?
soundness: 3
strengths: * The topic of automatically generating video datasets with accurate rationales is interesting and important for future multi-modality researches. It is very difficult to use human labor to fine-grain annotate videos with various reasoning details. 
* The authors release a large scale video instruction tuning dataset with rich CoT reasoning which significant saves the time and computing resources for future works.
summary: Large Multi-modal Models have achieved great success recently. However, when the problem comes to improve the reasoning capabilities of LMMs, the first and huge challenge is to get large scale and high quality datasets. Especially for video modalities, the problem is more hard by the difficulty and time-consuming. In this paper, the authors proposed Video-STaR, which is a method to generate video datasets with rationales based on LMMs. Besides, the authors releases a new video datasets named VSTAR-1M generated by the method.
weaknesses: * Limited technical novelty. The idea of this paper is very similar to STaR and all the proposed modules including answer generation, label rationalization and label verification are processed only for text modality, with no special considerations for video modality. It means that it is easy to transfer from normal STaR. Besides, for the methods section, the authors description is not clear and it is better to write it with more details
* Lack of experimental description, theoretical analysis, or other deeply analysis of some assumptions. In the paper, the authors claim that Label Rationalizations have higher possibility to result in hallucination. However, the authors don't provide further analysis, nor do some experiments to prove it (even only on some partial samples and do significance testing). Rather, the authors propose a case to show the problem in Figure.3. However, in cycle 1 I think the problem lays in LMMs repeats the questions instead of hallucination, and in cylce 2, Answer Generation also get a non-perfect result "with an overall score of 64.68" as the ground truth is 65.6

---

confidence: 3
contribution: 3
initial_rating: 5
presentation: 3
questions: 1. Recent studies indicate that using data generated by large language models (LLMs) for self-training can ultimately have a catastrophic impact on model performance. How do the authors address this issue? An interesting experiment could involve alternately training two models—Video-LLaVA (Model A) and another model (Model B)—using the Video-STaR strategy. In this setup, results generated by Model A could fine-tune Model B, and subsequently, results from Model B could fine-tune Model A. Observing the outcomes could reveal any differences in performance and stability under this iterative training approach.
soundness: 2
strengths: 1. This paper addresses a significant issue: how to utilize existing video dataset labels to improve instruction tuning. This paper presents the first self-training pipeline that could inspire future approaches in this area
2. The writing is clear. The authors also conduct extensive experiments to validate the proposed Video-STaR approach.
summary: This paper addresses the challenge of leveraging existing video datasets with diverse non-instruction labels to enhance video instruction tuning. The authors introduce Video-STaR, an automated loop for instruction generation, filtering, and self-tuning. Notably, they incorporate a Label Rationalization step within the loop to backtrace the rationale from video labels on challenging data. Experiments demonstrate that Video-STaR enables Large Multimodal Models to adapt to new tasks effectively.
weaknesses: 1. **Missing Experiments**: There is a lack of validation on the effectiveness of label filtering, including an ablation study on the verifier and clarification on why ground-truth (GT) labels should be included in the answers.
2. **Heuristic and Inflexible Label Verification**: The label verification process is heuristic, limiting flexibility and scalability. Since different videos contain diverse labels, the proposed approach requires custom parsers and verifiers for varying label formats. For example, timestamps may need distinct regex patterns due to format variations. Additionally, label matching, such as “chopping wood” vs. “smashing,” as seen in Table 2, appears to demand considerable human effort.
3. **Weak Baseline in Video-LLaVA Gemini**: Video-LLaVA Gemini is a relatively weak baseline. In Table 4, the authors used Gemini-pro to label only 2K samples for fine-tuning, a notably smaller dataset than those used for other baselines. The choice of 2K samples is not straightforward. The authors should attempt to label more videos with Gemini and include a curve showing the relationship between labeled data quantity and fine-tuning performance. How much labeled data is needed to achieve results comparable to Video-STaR’s 550K in both Table 3 and Table 4?
4. **Marginal Cross-Dataset Generalization (Table 4 & Table 7)**: Video-STaR’s improvement in cross-dataset generalization appears marginal. The authors attribute this to noise within the benchmark itself. If this is the case, comparisons and ablation studies across these four datasets may not convey valuable information. The authors could consider using alternative high-quality datasets to better evaluate Video-STaR’s true effectiveness in enhancing cross-dataset generalization.
5. **Typos**: Table index is incorrect. Line 260 Table 2.3 --> 2. Line 306 Table 3.2 --> 2

---

confidence: 3
contribution: 3
initial_rating: 5
presentation: 3
questions: * In Tables 6 and 7, the last row "- Generation" means that neither generation nor rationalization are used, or only that generation is not used while rationalization is used? Either way, it is important to see both experiments to better understand the contribution of each method.
* When doing self-training in multiple cycles, every time the model is initialized from the same non-instruction-finetuned checkpoint. Why not reuse the model from the previous cycle of self-training as initialization? 
* How do the authors deal with numerical labels such as temporal localization, bounding box, performance score? Do they expect the LVM to predict those directly and compute the L1 distance in the verifier? If so, how accurate is the prediction of the model in this case? Is Answer Generation capable of generating good question-answer pairs or Label Rationalization is more helpful here?
soundness: 2
strengths: * The idea of turning existing labeled video datasets into instruction tuning dataset and use them for self-training is elegant and effective.
* The paper is well written and is easy to follow
* I appreciate the ablation study
summary: The paper proposes a novel method to perform instruction tuning of Large Multi-modal Models (LMMs) on labeled video datasets, where the label format is arbitrary and different from that needed for instruction tuning. The main idea is to use the original LMM itself to generate question-answer pairs for instruction tuning, while using the original dataset labels as a "verifier" to filter out low-quality generations. The authors perform multiple cycles of dataset generation and model training to obtain the final model Video-STAR. Video-STAR shows improved performance on TempCompass and zero-shot video QA datasets.
weaknesses: * The main weakness of the paper is the Video-LLaVA baselines, i.e. Video-LLaVA+ and Video-LLaVA-gemini, which are the main points of comparison in the paper. Video-LLaVA-gemini has been fine-tuned only on several thousand sample pairs, which is incomparable to hundreds of thousands of video samples used for Video-STAR. As for Video-LLaVA+, the number of training samples is comparable, yet there is no information about how those samples were obtained. The way the samples in Video-LLaVA+ are constructed from the original video labels would affect the final model performance a lot, so it is important to explain this in details.
* Additionally, one important baseline for dataset creation is missing. It looks like the SoTA method for creating instruction datasets from video is VideoInstruct [A]. I would expect the authors to compare to this method of dataset creation, both in answer generation and label rationalization settings, to understand how much the video stream actually helps.
* It is somewhat unclear from the paper if Label Rationalization can be used to generate the entire dataset, without using Answer Generation. Somewhere in the paper the authors suggest that it leads to hallucinations, yet in Tables 6 and 7 they show the improvement.  

This is a preliminary rating and I will revise my score once the weaknesses and questions are addressed.

[A] -  Maaz et al. "Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Models"

Meta-review: This paper proposes a self-training approach for instructing tuning MLLMs with video.  

The reviewers find the approach interesting and also appreciate the dataset release.  

Reviewers raised questions and concerns primarily with the experimental details and qualified baselines.  These primarily addressed by the author responses.  

In the end, the paper received three positive reviews (accept, two borderline accept) and one borderline reject (AVhL).  During discussion, the reviewer AVhL think the paper can also be bumped up to borderline accept.   

The AC agrees with the reviewers in the interesting nature of the work and recommends that the paper be accepted.  It will make for a nice contribution to ICLR.

Decision: Accept (Poster)