Video-STaR: Self-Training Enables Video Instruction Tuning with Any Supervision
Authors: Orr Zohar
Xiaohan Wang
Yonatan Bitton
Idan Szpektor
Serena Yeung-Levy
Venue: ICLR 2025 Poster (2025)
Keywords: Video Understanding
Visual Instruction Tuning
Self-Training
Chain-of-thought reasoning
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: Introducing the first self-training method for video-LMMs

Abstract: The performance and reasoning capabilities of Large Multi-modal Models (LMMs) is dependent on the size and quality of their training datasets. However, collecting datasets that support chain-of-thought instruction tuning is highly challenging. Existing video instruction tuning datasets are often derived by prompting large language models with video captions to generate question-answer pairs, which makes them predominantly descriptive rather than reasoning-focused. 
Meanwhile, many labeled video datasets with diverse labels and supervision exist -- however, we find that their integration into LMMs is non-trivial. 
Herein, we present $\underline{\text{Video}}$ $\underline{\text{S}}\text{elf}$-$\underline{\text{T}}\text{raining}$ $\text{with}$ $\underline{\text{a}}\text{ugmented}$ $\underline{\text{R}}\text{easoning}$ (Video-STaR), the first self-training approach for video instruction tuning. 
Video-STaR allows the utilization of *any* labeled video dataset for video instruction tuning.
In Video-STaR, an LMM cycles between instruction generation and finetuning, which we show (I) improves general video understanding and (II) adapts LMMs to novel downstream tasks with existing supervision. 
During instruction generation, an LMM is prompted to propose an answer.  The answers are then filtered only to those that contain the original video labels, and the LMM is then re-trained on the generated dataset. 
By training exclusively on generated answers containing the correct video labels, Video-STaR leverages these existing labels as weak supervision for video instruction tuning.
Our results demonstrate that Video-STaR-augmented LMMs achieve notable improvements in (I) general Video QA, where TempCompass performance improved by 6.1%, *and* (II) downstream tasks, with a 9.9% increase in Kinetics700-QA accuracy and a 4.0% improvement in action quality assessment on FineDiving, while also exhibiting better interpretability.

Published as a conference paper at ICLR 2025
VIDEO-STAR: SELF-TRAINING ENABLES VIDEO IN- STRUCTION TUNING WITH ANY SUPERVISION
Orr Zohar"; Xiaohan Wang', Yonatan Bitton", Idan Szpektor & Serena Yeung-Levy"l
Stannford University. GGoolle Research foorrzohar, ayyeungggsstnnfoodd..ttu
Project page: http:s//orrrrooa.........o.oooor
ABSTRACT
The performance and reasoning capabilities of Large Multi-modal Models (LMMs) is dependent the size and quality of their training datasets. How- ever, collecting datasets that support chain-of-thought instruction tuning is highly challenging. Existing video instruction tuning datasets are often derived by prompting large language models with video captions to generate question-answer pairs, which makes them predominantly descriptive rather than reasoning- focused. Meanwhile, many labeled video datasets with diverse labels and supervision exist - however, we find that their integration into LMMs is non-trivial. To address this, we introduce Video Self-Training with augmented Reasoning (Video-STaR) the first self-training approach for video instruction tuning. Video-STaR allows the utilization of any labeled video dataset for video instruction tuning. In Video- STaR, an LMM cycles between instruction generation and finetuning, which we show (I) improves general video understanding and (II) adapts LMMs novel downstream tasks with existing supervision. During instruction generation, an then filtered only those that contain the original video labels, and the LMM is then re-trained on the generated dataset. By training exclusively generated answers containing the correct video labels, Video- leverages these existing labels as weak su- pervision for video instruction tuning. Our results demonstrate that Video-STafe STaR- augmented LMMs achieve notable improvements in (I) general Video QA, where TempCompass performance improved by 6. 6.1%, and (II) downstream tasks, with 9.9% increase in Kinetics700-QA accuracy and a 4.0% improvement in action quality assessment on FineDiving, while also exhibiting better interpretability.
LMM is prompted to propose answer. The answers
INTRODUCTION
The advent of Large Multi-moda: Models (LMMs) marked significant milestone in artificial intelli- gence. These models aim to create versatile systems capable of understanding and executing vision- and-language tasks aligned with human intentions. Liu et al. (2023a) demonstrated the importance of visual instruction tuning on the resulting LMM's performance. While significant advancements have been made with image-based LMMs, video-LMMs continue to face challenges due to the increased complexity of videos, which involve more intricate scene dynamics and temporal infor- mation. This complexity demands larger, more diverse, and peasoning-focusee tuning datasets. However, the largest existing video instruction dataset, Videolnstruct-100.. (VI- 100K) (Maaz al.) 2023). comprises 100K video-text pairs but only 13K unique videos. This is small compared to image instruction datasets like Cambrian- 10M (Tong et al. 2024). which contain millions of image question pairs from diverse domains and tasks.
video instruction
Furthermore, due to video instruction tuning dataset construction mainly prompting large language models to produce question-answer pairs- - these video datasets often degrade to simplistic questions, prompting for video captions - 75% of VI-100K's questions are of this type (see App. Fig. 10) lacking diversity and reasoning. Combining different sources supervision has the potential to generate more diverse video instruction tuning datasets, enhancing video understanding, Such pervision exists, as the broader computer vision community has developed an extensive collection of video benchmarks tailored for diverse tasks such as action recognition (Smaira al. 2020 Soomro

---

Published as a conference paper at ICLR 2025
Input: Video with label/labels
Instruction tuning
Output: Instruction Tuning
Question
Answer CoT
Question: What
is the action ...?
Text
BBOX
TAL
AR/AQA
Answer: first the bending....
Answer Generation
Label Verification
First, we see the man hending down lifting book...
First, we see the man bending down and lifting a book. Then, Finally, the man can be seen reading a book,
Label Down
What is the action sequence video?
Read Book
Figure I: Video-STaR Overview. Video-STaR can utilize any labeled video dataset, including AR (Action Recognition), AQA (Action Quality Assessment), and TAL (Temporal Action Localization) from which it generates video instruction tuning data (video, question, answer triplets). Internally, Video-STaR cycles between: (I) Answer Generation, where an LMM is prompted to generate candidate answers for the questions. (II) Label Verification where generated answers are filtered to only those that contain the video labels. And (III) Instruction Tuning, where a model is retrained on answers that pass verification. These cycles continue until performance plateaus, producing both an instruction tuning dataset and an improved LMM.
et al. 2012), action quality assessment (Xu et al. 2022b Zhang et al. 2023c). among others (Wu ct al.
2021 MeLughln et al., 202 Grauman et al. 2023 Luo et al. 2022).
Beyond improving overall LMM performance, adapting LMMs to novel or out-of-domain tasks is also crucial. While LMMs have many novel and impactful applications, many remain out of reach such analyzing radiology images (Senkaiahliyan et al. (202),, meteorological data (Lawson al. 2024). traffic analysis (Zhou & Knoll 2024), judging sporting events (c.g., gymnastics, Olympic diving), and assisting in surgical procedures, among others (Zheng et al.) 2024 Jiang et al. 2024
Deng et al. 2024). These tasks require expert, in-domain knowledge that LMMs lack, necessitat- ing adaptation through instruction tuning. However, collecting video instruction tuning datasets is complex and requires extensive manual effort. For instance, training an 'Al judge' to judge Olympic diving would traditionally involve collecting detailed expert critiques of each dive. On the other hand, these tasks often include auxiliary annotations that could be leveraged, such as surgical out- comes in medical procedures or judging scores in Olympic events.
To address these challenges. we take inspiration from LLM's capability for self-improvement (a.k.a self-training) (Huang et al., 2022 Zelikman al. 2022 (Sngh et al. 2023). which involves training model on its generated data and filtering to exclude low-quality outputs. A model's performance improved by cycling between generation, filtering, and training. Inspired by the success of self- training in LLMs, we hope that applying self-training to LMMs could improve their performance on complex reasoning and multi-modal tasks. Herein, we explore self-training in LMMs and introduce Video Telfnirainnn with augmented Reasoning (Video-STaa incorporation of any labeled video dataset of any format or task by prompting an LMM with the video and question to generate answers (Fig. 2 2. containing the video content's 's label, allowing the utilization of these labels as form of weak supervision. If the model cannot correctly answer the question, we provide the original video label and ask it to rationalize it (Fig. 2 2.2). We then again reject answers that do not contain the gold label (Fig. 2 2.3). By facilitating the use of any supervision for video instruction tuning, Video-STaR enables the creation of diverse datasets with thain-of-thoupe reasoning.
see Fig, [ Video STaR enables the
Our experimental setup initializes Video-STaR with Video-LLavA Lin al. 2023). focusing on assessing its impact on video question-answerin (VQA) performance. After a few Video-STaR training cycles, we compare the performance of Video to other LMMs and strong baselines, which utilize the entire source datasets, to gauge the effectiveness of the Video-STaR framework. Our findings demonstrate notable enhancements in accuracy and reasoning capabilities, highlighting Video-STaR's role in overcoming the constraints posed by conventional video instruction tuning datasets. We show that the integration of Video-STaR not only boosts Video-LLaVA's performance on standard zero-shot VQA benchmarks but also significantly improves its adaptability to various downstream video understanding tasks. This underscores Video-STaR's capacity to advance LMM training while improving overall performance and versatility.
2

---

Published as a conference paper at ICLR 2025
Our contributions be summarized as follows:
1. We introduce Video Self-Training with augmented Reasoning (Video-STaR), the first video self- self-training method for Large labeled video dataset for visual instruction tuning with chain-of-thought reasoning.
Models. Video-STaR enables the use of any
2. Video-STafe improves zero-shot video question answering performance on various bench- mark datasets, compared to strong baselines, as evidenced by increased accuracy on Tem- pCompass from 45.7% to 51.8%.
3. We demonstrate that Video-STaR can adapt LMMs diverse video tasks, notably enhanc- ing action quality assessment accuracy on FineDiving from 17.6% to 21.6%. Additionally, with Video-STaR, we show that LMMs can also explain why score is given.
4. Utilizing Video-STaR, we create a large, LM video instruction tuning dataset with rich sfain-of-thoughi reasoning VSTAR-IM, sourced from diverse datasets and tasks, and show that it benefits LMM performance.
2 VIDEO SELF-TRAININ WITH AUGMENTED REASONING (VIDEO-STAR)
Given a dataset of videos v and their corresponding labels = (1); Video-STaR's objective is create question q and chain-of-thouge answer a, that contains both the rationale and final answer (a = r Ul). These pairs are then used to instruction-tune the pre-trained model M on the dataset D {u,, qı, d)) producing the instruction-tume need not be from the same task, and may contain multiple labels. We start by prompting a large language model with task description T and video labels L to generate candidate questions q:
model M. Note that videos
Yr,L video labeled {L} for the task of {T}. What questions could you ask someone about the video that should contain the video labels in the response?
Video-STaR, performs ceterattio-ttaannig cycles, where in cycle i the instruction-tune model M is produced, while the instruction-tun tuned model from the previous cycle (i 1)+ is utilized for train- ing data generation. We initialize the process with 0 an existing instruction-tunes model.
To prepare the training data in cycle i, answers are generated either directly via Answer Generation or through backward rationalization via Label Rationalization. Answer Generation, (- prompted with questions (Sec. 2. Candidate answers are then filtered using the original video labels (Sec. 2.3). Videos rejected during direct Answer Generation are rationalized, where M (i-1) is provided both a video o; and labels 4, and then prompted with the question again (Sec. 2.2 Candidate answers are filtered again, creating the instruction tuning dataset in cycle i, D, of size d,. A pre-trained model M is then finetuned on D;, producing M The next cycle generates data using M until the performance plateaus (see Fig.
2.1 ANSWER GENERATION
Each Video-STaR cycle begins in direct Answer Generation. In this phase, (--1) is prompted with the video-question pair to provide an answer along with detailed rationale:
Ye= Question: (Q). Rationalize your answer step-by-step; how can one arrive at this conclusion?
When prompted with the question & on a particular video, A (f-1) expected to generate an answer a; that contains the label 1 and the rationale r; (o, = r; U 1, see Fig. 2) As can be seen in App Fig. [3 12 answers containing the correct labels are of higher quality and suffer less from hallucination. Therefore, we filter the generated answers to include only those that contain the correct label utilizing the verifier (Sec. (.3 For example of Answer Generation, see Fig.
2.2 LABEL RATIONALIZATION
Answer generation has two main drawbacks: (i) In some applications, especially on challenging/out- of-domain tasks, initial Answer Generation yield is low, resulting in almost training samples after filtering (c.g., FineDiving, see Fig. B]; (ii) improvement plateaus as the model fails to solve new problems in the training set, and it is only trained on examples it answers correctly.
3

---

Published as a conference paper at ICLR 2025
(2.1) Answer Generation
(2.3) Label
(2.2) Label Rationalization
Filtering
Video
Rationali-
Video
Rationale
Label
Question
zation
Question
Label
incorrect
Label
(+)
Label
Label
Label
correct
Repeat
(2) Instruction Tuning
Label
incorrect
Video
(2.3) Label
Question
Answer
Filtering
Video Self- Self-Traninn with augmented Reasoning. 2.1 We initialize by prompting an LMM to generate an answer for particular video. (2.3) We then filter the generated answers to those only containing the original video labels. 2.2 The videos whose generated answer did not contain the ground-truth labels are then sent to label rationalization, where given the video, question, and label model expected to rationalize the label. (2.3) The generated answers are filtered again to those only containing the ground-truth labels, and ( the I.MM is instruction-tuned from the pre-trained checkpoint on the resulting dataset. The cycle is then repeated.
Figure 2:
Inspired by Zelikman et al. (2022), for videos whose (i generated answers did not contain the ground-trute -uuut labels, we introduce label rationalization as part of Mideo-STaR. Concretely, provide M (f-1)* the video, question, and video label and instruct the model to rationalize the label:
Yo,r Question: (Q}. Answer: (L].
Can you rationalize the answer step-by-step? How can one arrive at this con- clusion?
Given the correct label, the model can reason backward and more easily generate rationale leading to the correct answer. However, Label Rationalization is more prone to hallucinations, so we prefer direct Answer Generation and use rationalizations only if answer generation fails (see App. Sec. F). For an example of Label Rationalization, see Fig. 3 The generated answers are then filtered, keeping only those that contain the gold label (i, = 1,) utilizing the verifier (Sec. 2.3 Label Rationalization is only utilized in training eycles; only Answer Generations produce the final model M
2.3 LABEL VERIFICATION
Video-STaR aims to utilize the labels weak supervision in instruction tuning data generation. Gold labels are a grounding aspect of our datasets and represent some ground-truth knowledge. In App. Sec. F.I show that answers that contain the ground-truth labels in their responses are of higher quality than those that don't 't and have higher probability of being correct. While we would like validate the existence of the different labels the generated text, this can be non-trivial.
To this end, we introduce the Verifier. The Parser, P extracts the predicted labels from the generated text (7, P(a;)), using a mixture of named entity recognition and Regex. Regex is used to identify easily identifiable string patterns, such as bounding boxes and time ranges, while named entity recognition is used for more nuanced entities, such timestamps. The Verfier, V compares
the extracted labels with the gold ones using the appropriate metries (V (V,4,,) loU for bounding boxes/temporal action localization, and BERT (Devlin et al. 2018) embedding similarity for sentence ordering. Each video has between 1-3 associated labels. To be classified as correct, the predicted labels must be within a 5% margin of error from the gold. See App. Tab, for a comprehensive list of the label types, their description, and corresponding parsers and verifiers.
R). For example,
3 VIDEO-STAR GENERATED DATASET VSTAR- M
In this section, we detail the different source datasets utilized our study (Sec. 3.1 and analyze the generated Video-STaR Dataset, VSTAR- 1M (Sec. 3.2

---

Published as a conference paper at ICLR 2025
Source
Videos
Labels
Avg. Dur.
Source Task
Kinetics700 (2020)
650K
700
5.1
Action Recognition
STAR
Video Reasoning, Temporal
benchmark
22K
207
28.6
Action I Localization, Bounding Box
2021)
FineDiving 2022b
Action Quality Assessment, Action Sequence
3K
1065
3.2
Table 1: Source Dataset Summary. Video datasets used as source datasets for instruction tuning data generation, their source tasks, and the number of unique labels.
Label Rationalization
[eppt Instruction| Please
performer's skill level action's toughness the video scale 1-100. Auswer.Action Action Sequenee: Inward, 3.5 Soms. Tuck. Eniry. Action difficult: 3.2. Overall Score: 65.6.
Auswer:
Can you explain step step-by-see step one
conclusion?
The performer in the vidco exccuies scries of flips and bwists the air, showcasing their chleticism and skill. The first third flip tuck...
flip 3.5 which is the final maneuver before the performer enters the water. Ovverall, the performer demonstrates high level of throughout ruutine, making challenging impressive display acrobatics.
flip
somersault...
entry.
Answer Generation
Cycle
First, we need identify the action soquence performed by the diver... inward somersaull, followed by a somersault into the Next. we need to assess the difficulty each action the sequence.
this case, diffficully cach action follews: Inward: 3.5, Soms Tuck: 3.2, Entry: 3.0... this case. the average difficulty score 3.2... conclude that the diver's action sequence high skill level and is quite tough, with in overall difficulty score 65.6)
tack, then
Cycle
Answer Generation
Finctuning
[Input Instruction/ Please assess the performer's
loughness video scale
Can explain step st- how one can artive at this conclusion?
First, we need identify the action sequence performed by the diver. In this case, the diver performs an inward somersmii tuck, to a 3.5 somensault tuck 3.2...
somersauli, followed
entry water.. action difficulty score
Cycle 2
action sequence high
quite tough,
[Label] 65.6
Qualitative Improvement of Data Generation over Cycles FineDiving. We initial- ize the model with Video-LLaVA (Cycle O), where the model cannot generate an answer (++ |x) or rationalize the label correctly () x). In the second cycle (Cycle 1), the model still cannot gener- ate an answer (+ xx) but can rationalize the video label (VI-) +). which is selected for instruction tuning. Finally, which is selected for visual instruction tuning. We highlight in green correct answers, in red wrong answers, and in yellow hallucinations.
Figure 3:
the third cycle (Cycle 2), the model directly generates a correct answer (V) -t),
3.1 SOURCE DATASETS
In selecting source datasets, we selected datasets that contain diverse video content and label types, please see Tab. []] These include Kinetics700 (Smaira et al. 2020). which has action recognition annotations and is particularly large and diverse. FineDiving (Xu et al., 2022b) is an action quality assessment dataset of Olympic diving events and has both an overall score and action sequence annotations. Finally, STAR-benchhin bounding box and temporal action localization annotations. Tab. [ contains the relevant dataset statistics, e.g., the number of videos and labels per dataset.
(Wu et al., (02I). a video reasoning dataset, also contains
3.2 GENERATED DATASET ANALYSIS
Quantitative Analysis. Through the application of Video-STaR, STaR, significant dataset augmentation was achieved two cycles of, illustrated in Fig. This figure displays the Answer Generation
5

---

Published as a conference paper at ICLR 2025
Kinetics700
STAR-Benchmark.
FineDiving
Video
On scale from 1-100, how would you evaluate the execution of the actions the video!
What type activity is being videa?
Which object was pur down by the person?
Question
the viden, the picking up a bag and putting down the floor. After that, be
the viden, a man seen holding large axe and swinging around while crowd watches. He procodds to hit a block of wood with Based these observa- tions, it can he concluded that the activity chopping wood.
First, we identify the tion sequence: inward somersault entry
ac-
water. Evalunt- ing each movement's difficulty, we assign ioward somersault: 2.5, s... By averaging these scoces ...
picks up a book and puts it down the floor. Based on this informa- tion, can conclude that the per- son pat down the bag and picked up book.
Generated Answer
the overall scure 63.0 out of 100.
Label
Smashing
The bag.
Overall Score 63.0
Examples of Generated Data. Examples of the video, question, Video-STaR generated answer, and ground-truth label from each source dataset.
Table 2:
Kinetics70p
Benchmark
FineDiving
Answer Generation
Label Rationaliatton
Generation
Label Rationalization
25
Answer Generation
Label Rationalization
Cycle
Cyde
Cyde
Figure 4: Dataset Yield vs. Cycles. Percentage of the videos converted to instruction tuning by the Answer Generation and Label Rationalization per dataset. As can be seen, on difficult datasets, such as FineDiving, no videos are converted by Answer Generation in the first cycle. By utilizing Label Rationalization, the model is able to improve eventually generate answers correctly.
and Label Rationalization yield across the datasets source. Notably, the initial application of Video- LLaVA datasets like Kinetics700 and STAR-Benchnark showed significant Answer Generation success rates. However, the FineDiving dataset presented a notable challenge, with Answer Gen- eration having no answers generated directly, underscoring the complexity of the dataset and the critical role of Label Rationalization. By the end of the second cycle, a substantial number of high- quality instances had been produced, showcasing both the effectiveness of Video-STa in converting labeled video datasets into video instruction tuning dataset, as evidenced in Fig.
Qualitative Analysis. See Tab. 2 for examples of generated question-answer pairs. From Kinet- ics700, we extracted an instance showcasing a video labeled 'smashing'. Video-STaR correctly identified a more fine-grained label, 'chopping wood'. the FineDiving dataset, a clip depicting complex dive was accompanied by the question 'On a scale from 1-100... The model's output text provided a breakdown of the dive's s components, leading to a score (label), as would be desired from LMM visual assistant. Finally, in the STAR benchmark. questions already provided; therefore, we utilized them directly.
In we show the qualitative improvement of the generated data over Video-STai cycles. In the first cycle (Cycle 0), Video-LLaVA failed at Answer Generation and Label Rationalization. After one Video-STaR cycle (Cycle 1), Video-STaR still failed Answer Generation but succeeded in Label Rationalization. After the final Video-STaR cycle (Cycle 2), Video-STaR managed to generate the answer without requiring the label via Answer Generation.
4 EXPERIMENTS
We experimented with Video-STaR and evaluated its enhanced video understanding capabilities. In Sec. 43 we evaluate how Video-STaR adapts Large Multi-modal Models (LMMs) to the source datasets and how these capabilities are transferred zero-shot to similar benchmarks. Sec. 4.2 we evaluate the video question-answering capabilities video benchmark datasets.

---

Published as a conference paper at ICLR 2025
Action
Direction
Speed
Event Order
Attribute Change Both
Avg.
Fine
Coarse
Obj.
Cam.
Abs.
Rel.
Color
Size
Other
Random
39.7
40.1
39.8
39.0
40.8
42.0
41.5
40.4
39.9
38.9
39,4
40.5
mPLUG-Owl
2023
48.8
66.1
38.7
36.8
42.2
38.4
42.0
41.7
44.7
41.9
39.9
44.4
Video-LLaaA
2025
63.4
93.5
36.1
34.8
42.7
26.5
39.1
52.6
37.1
43.3
33.3
45.7
Video-I.IaVA
62.1
93.0
35.0
32.6
41.1
38.7
36.4
59.0
40.2
36.7
44.4
47.2
Vid-LLaVA Gemina
65.7
90.5
38.9
46.0
41.8
42.4
41.0
48.7
49.1
32.5
40.4
48.7
Video-STaR
68.6
94.1
39.9
39.0
40.7
43.0
41.1
53.8
48.5
45.0
55.6
51.8
Gemini-1.5 (024
94.8
98.4
43.6
42.4
65.3
48.7
55.6
79.5
59.8
70.0
66.7
66.0
Table 3: Comparison with state-of-the-art methods on TempCompass. TempCompass (Liu et al.
2024 assesses the temporal understanding capabilities of video language models across five dimen- sions Video-STaR improves Video-LLaVA performance on TempCompass by 5%.
Dataset size
MSVD-QA
MSRVTT-QA
TGIF-QA Accuracy
ActivityNet-QA
Methods
Accuracy
Score
Accuracy
Score
Score
Accuracy
Score
VideoChat (2023b
4K
56.3
2.8
45.0
2.5
34.4
2.3
2.2
Video-LLaMA
(2023a
4K
51.6
2.5
29.6
1.8
12.4
1.1
Video-ChatGPP
2025
100K
64.9
3.3
49.3
2.8
51.4
3.0
35.2
2.7
Video-I.LaVA
2023
100K
69.7
3.9
57.4
3,5
46.5
3.3
43.2
3.4
Video-LLaVA +
650K
67.8
3.8
56.0
3.4
46.5
42.2
3.3
Vid-LLaVA VA Geraini
128K
67.2
3.9
56.9
3.4
45.5
3.4
42.6
3.3
Video-STaR
550K
71.3
4.0
58.2
3.5
47.3
3.3
43.2
3.3
Gemini- 1.5-pro 3-ro (2024)
71.6
3.9
52.6
3.2
45.0
3.1
56.7
Table Zero-shot Video QA benchmarks. As can be seen, many models are approaching Gemini performance indicating that LMMs may be operating near the noise level on these benchmarks.
4.1 EXPERIMENTAL SETTING
Implementation Details. We initialize from the Video-LLaVA (Lin al., 2023) model, which utilizes the Vicuna-7B v v1. (Chiang et al., 2023). We ran three Video-STaR cycles, and each cycle was initialized with the pre-trained Video-LLav batch size, AdamW optimizer, and cosine learning rate schedule. The learning rate is 2e - 5 with 0.03 warmup ratio. In combination with the generated Video-STaR instruction tuning dataset, we additionally utilized the Videolnstruct-1000 (Maaz et al. (023) and the LLaVA v1.5 instruction tuning datasets (Liu et al. 2023a). Additional details are available the appendix.
weights. We train for one epoch using 128
Baselines. Besides comparing to Video-LLaVA, we also wanted to evaluate the effect of utilizing additional data and naively adapting the source datasets. Therefore, utilized simple templates to generate question-answer pairs from the video labels and trained Video-LLaVA on the resulting dataset. We will reference this baseline Video-LLaVA+. Another baseline for adapting Large Multi-modal Models to novel tasks is model distillation, where a stronger video model in this work, Gemini 1.5 pro-vision dataset) and used to finetune the models. Unlike Video-LLaVA' we initialized from the fine-tuned Video-IL.a VA model and mixed the dataset to mitigate forgetting. We will reference this baseline as Vid-LLaVA Gemini Veninii
utilized label/annotate small of videos (3500 from each
dataset with 50% of Video-LLaVa original
Evaluation Details. We evaluate on the following benchmarks; the Zero-shot question-answer (QA) benchmarks: MSVD-QA, MSRVTT-QA. TGIF-QA, and ActivityNet-QA (Xu al., 2017
2016 Jang et al. 2017: Heilbron et al., 2015). TempCompass (Liu et al. 2024), multipl--hooie fin-grainee QA benchmark. Adapted task performance is evaluated by converting source datasets using simple templates and applying the same evaluation protocol as Maaz et al. (Maaz et al.) 2023), and FineDiving-
This protocol reports two metrics: accuracy (the percentage of correctly answered questions) and the average score (where ChatGPT rates each response on a scale 1-5 and calculates the mean of these scores). All evalu- ations utilize the same GPT model (Wu) 2024) ("gpt-3.5-turbo") to ensure consistent comparisons. Due to cost considerations, 1000 videos were randomly selected from each dataset for Gemini eval- uation. The reported values used Act.vitlyNet-QA.
producing Kinetics700-QA.
TTAAR-ennchark-O00

---

Published as a conference paper at ICLR 2025
Kineties700-QA
STAR-hench-QA
FineDiving-
Methods
Accuracy
Score
Accuracy
Score
Accuracy
Score
Video-LLaVA
50.0
3.2
24.9
2.6
17.1
2.2
Video-LLaVA
49.5
3.2
28.8
2.8
17.6
2.2
Vid-LLaVA Gemini
49.4
2.7
29.3
2.6
16.5
Video-STaR
59.9
3.5
33.0
2.9
21.7
2.3
Table 5: Adapted Dataset Performance. Performance metrics test sets of Kinetics700, Fine- Diving, and STAR-benchnark datasets via converting them to QA following Maaz et al. (2023).
Video-STaR shows significant improvement over Video-LLaVA and Video-LLaVA" showing the potential of Video-STaR for LMM adaptation to new tasks.
4.2 QUANTITATIVE EVALUATION ON ZERO-SHOT BENCHMARKS
effect general video question answering, evaluated its effect L''s'sLLaVA's performance on TempCompass, see Tab. 3 On TempCompass, Video-STaR out- performed Video-LLa LLaVA across the board- by 5%. To see if this performance boost is simply factor of training on larger dataset, we also evaluated Video-LLaVA Video-LLaVA was trained on even a larger video dataset by naively utilizing video labels, and yields more modest improve- ment of 3%, showing the utility of Video-STaR. TempCompass is also a fine-grained dataset that would be sensitive hallucinations, indicating that Video-STaR is not more prone to hallucina- tions compared to existing methods. Gemini 1.5 pro scored an impressive 66.0 on TempCompass, showing there still much room for improvement on this benchmark.
To evaluate
We then continued and evaluated Video-STaR's effect zero-shot video QA performance on the MSVD-QA, MSRVTT-QA, TGIF-QA and ActivityNet-QA benchmarks. As can be seen Tab. 4 Video-STake achieves performance improvements where. for instance, the MSVD-QA dataset, Video-STaR attains the highest accuracy of 71.3% vs Video-LLaVA's 69.7. On MSRVTT-QA, Video-STaR leads with an accuracy of 58.2% and maintains competitive edge in other datasets like TGIF-QA and ActivityNet-QA. Seeing the relatively small performance gains compared to Temp- Compass, we additionally evaluated Gemini 1.5 pro- pisivnsion 1000 video subsets of each dataset and found that its performance is on par with existing open-source models. We believe this shows that we
the 'noise" limit of these benchmarks. Our qualitative analysis indicated that many of the questions selected as 'wrong' are actually due to the benchmark design-overs general ques- tions with multiple correct answers. Concurrent work (Wul (202) has similarly concluded that the ChatGPT-3.5 version utilized in evaluation can lead to variations of =10 in accuracy.
4.3 QUANTITATIVE EVALUATION ON ADAPTED DATASETS
Besides improving general visual question-answerinn -nswering performance, Video-STaR also adapt Large modti-modal models to novel takes. To demonstrate this, we converted the test sets (nor included in training) of the source datasets Kinetics700, STAR-benchmark, and FineDiving. The results of these evaluations are reported in Tab. 5)) Adapting LMMs with easier-to-collect labels can be helpful in various applications, leading more versatile, multi-domain capable assistant. When evaluating Video-STaR's 's impact on LMM performance on the diverse source datasets, we found that it significantly improves model performance, particularly complex tasks. For instance, on Kinetics700, known for its extensive action categories, Video- STaR enhanced Video-LLaVis performance accuracy by an average of 20% (as can be seen in Tab. 55) showcasing its ability to develop generalized models adept across multiple domains. Interestingly, Video-I.LaVA++ perfor- mance did not improve compared to Video-I.LaVA, and in some cases, even worsened, showing that cannot directly utilize labeled datasets for LMM adaptation.
Action Quality Assessment (AQA) is a complex video task requiring detailed action understanding, where Video-STaR significantly enhanced LMM performance the FineDiving dataset. Our results show a notable improvement from 17.6 to 21.6 in score prediction accuracy, highlighting Video- STaR's 's effectiveness in refining LMM's temporal reasoning. However, Video-STaR allows LMMs to not only rate a particular dive but also explain the rationale behind each assessment. This rationale is invaluable for many applications, effectively providing potential user feedback for improvement.
8

---

Published as a conference paper at ICLR 2025
Kinetics700-00
STAR-bench-QA
FineDiving-Qs
Ablations
Accuracy
Score
Accuracy
Score
Accuracy
Score
Vidco-STal
59.9
3.5
33.0
2.9
21.6
2.3
Generation 58.8
3.5
28.7
2.7
17.1
2.1
Rationalization 59.8
3.5
26.6
2.7
12.8
2.0
-Generation 50.0
3.2
24.9
2.6
17.6
2.2
Table Ablations on Adapted Datasets. Performance metries on test of Kinetics700, STAR- benchmark, and FineDiving datasets. Label Rationalization impacts mostly the difficult datasets, such FineDiving, whose initial Answer Generation yields low.
MSVD-QA
MSRVTT-QA
TGIF-QA
Netivi
Ablations
Accuracy
Score
Accuracy
Score
Accuracy
Score
Accuracy
Score
Video-STaRR
71.3
4.0
58.2
3.5
46.8
3.3
42,2
3.3
Generation
70.6
3.9
57.8
3.5
44.9
3.3
41.1
3.2
Rationalization
70.6
3.9
57.5
3.5
47.7
3.4
42.2
3.3
Generation 69.7
3,9
57.4
3.5
46.5
3.3
43.2
3.4
Table Ablations on Zero-Shot Benchmarks. In simpler benchmarks, Answer Generation proved more critical for sero-shot generalization than Label Rationalization.
This advancement enables novel applications, from sports coaching to automated feedback systems, by offering evaluations and constructive feedback. The ability to interpret and improve action quality underscores the potential Video-STaR, underscoring the potential of utilizing L.MSs as intelligent and informative visual assistants. For more, please App. Sec. B
4.4 ABLATIONS
In our ablation studies, we evaluated the impact of removing Label Rationalization and Answer Generation from Video-STaR, focusing on adapted datasets (Kinetics700, FineDiving, STAR- benchmark) and zero-shot benchmarks (MSVD-QA, MSRVTT-QA, TGIF-QA, ActivityNet-QA).
Adapted Datasets For adapted datasets (Tab. 4.3 excluding Label Rationalization led to a sig- nificant performance drop in FineDiving. from 21.6 to 12.8 in accuracy, highlighting its critical role in complex reasoning tasks. This is likely due to the lack of conversion of any examples from the data. However, the removal of Answer Generation resulted in more pronounced and uniform decline across all datasets. For example, Kinetics700" accuracy was reduced from 59.9 to 50.0, underscoring foundational role in generating context-relevant responses.
Zero-shot benchmarks In zero-shot benchmarks (Tab. 4.3) the removal of Label Rationaliza- tion had mixed impact, slightly affecting MSVD-QA where accuracy decreased from 71.3 to 70.6. The climination of Answer Generation consistently lowered performance, such decrease in MSRVTT-QA accuracy from 58.2 to 57.4. ActivityNer-QA performance improved, probably because Inntrutttt utilizes ActivityNet for instruction tuning. Therefore, the introduction of additional videos decreases performance.
RELATED WORKS
In Sec. 5.1 recent advancements in Large Vision-Language Models and video instruction tuning datasets are introduced. In Sec. 5.2, advancements Large Language Models and self-supervised instruction tuming are explored.
5.1 LARGE VISION-LANGUAGE MODELS
Initial LMMs, such as LLaVA (Liu et al. 2023b a) and BLIP-2 (Li et al. 2023a). demonstrated the potential of merging image inputs with large language models. Methods like mPLUG-Owl (Ye

---

Published as a conference paper at ICLR 2025
et al. 2023) and Flamingo (Alayrac et al. 2022) further allowed for multiple image inputs without architectural changes. Li et al. (2023b) and Zhang et al. (2024) led the transition to video un- derstanding. integrating video/image encoders and LLMs while training small video instruction tuning datasets. Jin et al. 2023) introduced Chat-UniVi, a unified model employing dynamic visual tokens for both images and videos, optimizing visual token usage and higher frame count sampling. LLaMA-VID (Li et al., (023c) showed that the token count can be further reduced by pooling the tokens selectively via the text prompt using Q-Former. Wang al. (2024b) introduced training- free agentic approach. Recently, Video-LLaVA (Lin et al. (2023) used modality -seeciii---piiiccce encoders for video and image inputs to leverage LanguageBind encoders as they are constructively aligned during pretraining and utilized shared projection.
Maaz al. (2023) expanded the field with the first large video instruction tuning dataset, Videolnstruct-In TOOK. VideoInstruct-100K was generated from ActivityNet (Heilbron et al., 2015) by prompting chatGPT with the video captions, generating question-answer answerr pairs. While driving much of the performance improvement in the field (Jin et al. 2023 Wang et al. 2023 Lin et al.
2023 Li et al. 2023c), upon examination of Videolnsttrutt-00K, it is evident that it suffers from quality issues. The questions often degrade into de facto prompts for video caption (see Fig. 10P and rarely require many spatiotemporal capabilities, which may limit LMM performance.
5.2 LARGE LANGUAGE MODELS AND SELF-TRAINING
The advent of GPT (Radford et al. 2018 Brown et al.
2020) marked significant milestones in natural language processing, showcasing LLMs' power in understanding and generating human-lit -ike
text. Open-source LLMs like LLaMA (Touvron et al.] 2023a bb and their instruction-tuned variants like Alpaca and Vicuna (Taori et al. 2023] Chiang et al. 2023) further tailored these models for nuanced human- Al interactions. However, even LLMs have found it challenging to scale annotated datasets for training, prompting work on self-training and self-immpovemment (Singh al. [023]
Huang et al. 2022 Ho et al.] 2023] Marasović et al.] 2022 Hosseini et al. 2024). In this line of work, L.Ms cycle between instruction-tunie
data generation and instruction tuning, iteratively improving LLM performance over cycles. For instance, Zelikman et al. (2022) introduced the the Self-Taught Reasoners method, used rationalization generate frain-of-thoughe (CoT) reasoning, filtering poor rationalizations to retain correctly answered questions. Other self-training approaches include expectation pnpectaaioo-maximizzaiin maximization-ban based approaches (Singh et al. 2023), which alternate between data generation and improvement between training cycles. Alternatively majority voting has also been utilized to generate answers and rationale for unlabeled questions (Huang al. (022).. These methods show the effectiveness of iterative self-training. In our work, we aim to introduce weakly supervised self oeff-trainnng approach for vidco instruction tuning, leveraging video supervision that is often easier to collect and exists in many large and diverse datasets.
CONCLUSIONS
In conclusion, Video Self-Training with augmented Reasoning (Video-STaR) presents a novel ap- proach to enhance Large Multi-Language Models (LMMs) by enabling the use of diverse labeled video datasets for visual instruction tuning. Video-STaR addresses critical data diversity and qual- ity challenges, leading to performance improvements across various vidco understanding tasks. Our experiments demonstrate Video-STaR's effectiveness in source dataset dataset adaptation and zero-shot gen- eralization, showcasing its potential in advancing LMM capabilities for complex video reasoning.
The promising results of Video-STaR open new research avenues, particularly in expanding LMM knowledge bases using readily available image and video datasets. Future work could explore ad- vanced self-training techniques and integration with emerging LMM architectures, focusing on long- form video understanding to boost LMM understanding. Additional work is also needed reduce hallucinations, perhaps by using grounded VLMs as agents to further supervise the generation.
Acknowledgements. We thank Google Research for providing financial support through Stanford HAI- MoA-Google collaboration. This work was partially supported by the National Science Foundation under Grant No. 2026498. We thank the Knight- Hennessy Scholars Foundation for funding OZ.
10

---

Published as a conference paper at ICLR 2025
REPRODUCIBILITY STATEMENT
All code, source files, and generated dataset text instructions can be found in our supporting infor- mation and will be made publicly available. Additional implementation details can also be found in the Appendix, Sec. A
REFERENCES
Man-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andrew Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals. Andrew Zisserman, and Karen Simonyan. Flamingo: visual language model for few-shot learning, 2022.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz. I. Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 1877-1901. Curran Associates, Inc., 2020.
Wei-Lin Chiang. Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph F. Gonzalez, lon Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL https:
7 ///_ys..rg//loo//22233 03 30- vicuna/
Adam Cohen. seatgeek/fuzzyywwzzyyy : Release 0810.,, February 2020. URL https::/
3xC//zojject/Eaayyzzz2
Jiawen Deng, Kiyan Heybati, and Matthew Shammas-Toma, When vision meets reality: Exploring the clinical applicability of gpt-4 with vision, 2024.
Jacob Devlin, Ming-Wos Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04800 04805, 2018.
Kristen Grauman, Andrew Westhury, Lorenzo Torresani, Kris Kitani, Jitendra Malik, Triantafyllos Afouras, Kumar Ashutosh, Vijay Baiyya, Siddhant Bansal, Bikram Boote, Eugene Byrne, Zach Chavis, Joya Chen, Feng Cheng. Fu-Jen Chu, Sean Crane, Avijit Dasgupta, Jing Dong, Maria Escobar, Cristhian Forigua, Abrham Gebreselasie, Sanjay Haresh, Jing Huang, Md Mohaiminul Islam, Suyog Jain, Rawal Khirodkar, Devansh Kukreja, Kevin Liang, Jia- Wei Liu, Sagnik Ma- jumder, Yongsen Mao, Miguel Martin, Effrosyni Mavroudi, Tushar Nagarajan, Francesco Ra- gusa, Santhosh Kumar Ramakrishnan, Luigi Seminara, Arjun Somayazulu, Yale Song, Shan Su, Zihui Xue, Edward Zhang, Jinxu Zhang, Angela Castillo, Changan Chen, Xinzhu Fu, Ryosuke Furuta, Cristina Gonzalez, Prince Gupta, Jiabo Hu, Yifei Huang, Yiming Huang, Weslie Khoo, Anush Kumar, Robert Kuo, Sach Lakhavani, Miao Liu, Mi Luo, Zhengyi Luo, Brighid Meredith, Austin Miller, Oluwatumininu Oguntola. Xiaqing Pan, Penny Peng, Shraman Pramanick, Merey Ramazanova, Fiona Ryan, Wei Shan, Kiran Somasundaram, Chenan Song, Audrey Souther- land, Masatoshi Tateno, Huiyu Wang, Yuchen Wang, Takuma Yagi. Mingfei Yan, Xitong Yang, Zecheng Yu, Shengxin Cindy Zha, Chen Zhao, Ziwei Zhao, Zhifan Zhu, Jeff Zhuo, Pablo Arbe- laez, Gedas Bertasius, David Crandall, Dima Damen, Jakob Engel, Giovanni Maria Farinella, An- tonino Furnari, Bernard Ghanem, Judy Hoffman, C. Jawahar, Richard Newcombe, Hyun Soo Park, James M. Rehg, Yoichi Sato, Manolis Savva, Jianbo Shi, Mike Zheng Shou, and Michael Wray. Ego-exo4d: Understanding skilled human activity from first- and hird-persoon perspectives. arXiv preprint arXiv:2311.18255 2023.
Nagaaaaa
Madeleine Grunde-MeLaught McLaughlin, Ranjay Krishna, and Maneesh Agrawala. Agqa: A benchmark for compositional apatio-temport reasoning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021.

---

Published as a conference paper at ICLR 2025
Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. Activitynet: A large-scale video benchmark for human activity understanding, In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 961-970, 2015. doi: 10.1109/CVPR.2015. 7298698.
Namgyu Ho, Laura Schmid, and Se- Young Yun. Large language models reasoning teachers. arXiv preprint arXiv:2212.10000 10071, 2023.
Arian Hosseini, Xingdi Yuan, Nikolay Malkin, Aaron Courville, Alessandro Sordoni, and Rishabh Agarwal. V-star: Training verifiers for self-taught reasoners, 2024.
Jiaxin Huang, Shixiang Shane Gu, Le Hou, Yuexin Wu, Xuezhi Wang, Hongkun Yu, and Jiawei
Han. Large language models can self-improve. ArXiv, abs/2210.11610. 2022. URL https:::
org Cttp//ooooIII::330000000 //api. www.ssematiicchhll.... htt.a//oo.aaa.oooooooooossaaaaa.
Yunscok Jang. Yale Song, Youngjae Yu, Youngjin Kim, and Gunhee Kim. Tgif-qa: Toward spatio- temporal reasoning in visual question answering. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 2758-2766, 2017.
Yixing Jiang. Jesutofunmi A Omiye, Cyril Zakka, Michael Moor, Haiwen Gui, Shayan Alipour, Scyed Shahabeddin Mousavi, Jonathan H Chen, Pranav Rajpurkar, and Roxana Daneshjou. Eval- uating general vision-language models for clinical medicine. medRxiv, pp. 2024-04, 2024.
Peng Jin, Ryuichi Takanobu, Caiwan Zhang, Xiaochun Cao, and Li Yuan. Chat-univi: Unified vi- sual representation empowers large language models with image and video understanding, arXiv preprint xrXiv:231..001 2023.
John R Lawson, Montgomery L Flora, Kevin H Goebbert, Seth N Lyman, Corey K. Potvin, David M Schultz, Adam J Stepanck, and Joseph E TrujilloFaalóón. Pixels and predictions: Po- tential of gpt-4v in meteorological imagery analysis and forecast communication. arXiv preprint arXiv:2404.15 2024.
Vladimir I. Levenshtein. Binary codes capable of correcting deletions, insertions, and reversals. So- vier physics. Doklady, 10:707-710, 1965. URL aittps://aptl http::///app..eemant.cccccoooooo org/
Corpus 60:2771:00000111 60827152
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. BLIP-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference Machine Learning, volume 202 of Pro- ceedings of Machine Learning Research, pp. 19730-19742. PMLR, 23-29 Jul 2023a. URL https: /ttpppeeooennssseepepeppppes..ee www.ggsmm.....ommm mlr press wwtp:/222/////222 w2tp//2220.........
Kunchang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. Videochat: Chat-centric video understanding. arXiv preprint arXiv:2305.06355 2023b.
Yanwei Li, Chengyao Wang, and Jiaya Jia. Llama-vid: An image is worth 2 tokens in large language models, 2023c.
Bin Lin, Bin Zhu, Yang Ye, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by alignment before projection. arXiv preprint arXiv:2311.10122, 2023.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. arXiv preprint arXiv:2310.03744 2023a.
Haotian Liu, Chumyuan Li. Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. arXiv preprint 2rXiv:2304.0844 2023b.
Yuanxin Liu, Shicheng Li, Yi Liu, Yuxiang Wang, Shuhuai Ren, Lei Li, Sishuo Chen, Xu Sun, and Lu Hou. Tempcompass: Do video llms really understand videos? arXiv preprint arXiv: 2024.
12

---

Published as a conference paper at ICLR 2025
Zelun Luo, Zane Durante, Linden Li, Wanze Xie, Ruochen Liu, Emily Jin, Zhuoyi Huang, Lun Yu Li, Jiajun Wu, Juan Carlos Niebles, et al. Moma-lrg: language-refine graphs for multi-object multi-actor activity parsing. Advances in Neural Information Processing Systems, 35:5282-5298, 2022.
Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models. arXiv preprint arXiv:2306.05424, 2023.
Ana Marasović, Iz Beltagy, Doug Downey, and Matthew E. Peters. Few-shot self- reffrrationalization with natural language prompts. arXiv preprint arXiv:2111.08284, 2022.
Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, al. Improving language under- standing by generative pre-training. 2018.
Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean- baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. Gem- ini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530. 2024.
Senthujan Senkaiahliyan, Augustin Toma, Jun Ma, An-Wen Chan, Andrew Ha, Kevin R An, Hrishikesh Suresh, Barry Rubin, and Bo Wang. Gpt-4v (ision) unsuitable for clinical care and education: a dliician-evvluatte assessment. medrxiv. Preprint posted online on November, 16, 2023.
Xiaoqian Shen, Yunyang Xiong, Changsheng Zhao, Lemeng Wu, Jun Chen, Chenchen Zhu, Zechun Liu, Fanyi Xiao, Balakrishnan Varadarajan, Florian Bordes, et al. Longvu: Spatiotemporal adap- tive compression for long video-language understanding. arXiv preprint arXiv:2410. 17.3444 2024.
Avi Singh, John D. Co-Reyes, Rishabh Agarwal, Ankesh Anand, Piyush Patil, Xavier Garcia, Pe- J. Liu, James Harrison, Jachoon Lec, Kelvin Xu, Aaron Parisi, Abhishek Kumar, Alex Alemi, Alex Rizkowsky, Azade Nova, Ben Adlam, Bernd Bohnet, Gamaleldin Elsayed, Hanie Sedghi, Igor Mordatch, Isabelle Simpson, Izzeddin Gur, Jasper Snoek, Jeffrey Pennington, Jiri Hron, Kathleen Kenealy, Kevin Swersky, Kshiteej Mahajan, Laura Culp, Lechao Xiao, Maxwell L. Bileschi, Noah Constant, Roman Novak, Rosanne Liu, Tris Warkentin, Yundi Qian, Yamini Bansal, Ethan Dyer, Behnam Neyshabur, Jascha Sohl-Dickstein, and Noah Fiedel. Beyond human data: Scaling self-training for problem-solving with language models, 2023.
Lucas Smaira, João Carreira, Eric Noland, Ellen Clancy, Amy Wu, and Andrew Zisserman. A short note on the kinetics-700-2200 2020 human action dataset, 2020. URL /ttp:::///eeemmmnee com/
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: dataset of 101 human actions classes from videos in the wild, 2012.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An cnsruction-followinn llama model. https: / //g thub. com wtt.:/ctootttae.eeeee ttttttttssuuttaeee 1ab/ stanford_ lpaca] 2023.
Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai Charitha Akula, Jihan Yang, Shusheng Yang, Adithya Iyer, Xichen Pan, et al. Cambrian-1: A fully open, vision-centric exploration of multimodal 1lms. arXiv preprint arXiv:2406. 16860, 2024.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Ar- mand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971, 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull. David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshom, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel
13

---

Published as a conference paper at ICLR 2025
Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 2023b.
Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. arXiv preprint mxXiv2249.121999 2024a.
Xiaohan Wang, Yuhui Zhang, Orr Zohar, and Serena Yeung-Levy. Videnagent: Long-form video understanding with large language model agent. European Conference Computer Vision (ECCV), 2024b.
Yizhou Wang, Ruiyi Zhang, Haoliang Wang, Uttaran Bhattacharya, Yun Fu, and Gang Wu. Vaquita: Enhancing alignment in llm-assisted video understanding. arXiv preprint arXiv:2312.02310, 2023.
Bo Wu, Shoubin Yu, Zhenfang Chen, Joshua B. Tenenbaum, and Chuang Gan. STAR: A bench- mark for situated reasoning in real-world videos. In Thirty-fithh Conference on Neural Infor- mation Processing Systems Datasets and Benchmarks Track (Round 2), 2021. URL hetps:
//penrevi ew www.nee/.frrmmt d=E ZA
Wenhao Wu. Freeva: Offline mllm training- frreeng-free video assistant. arXiv preprint arXiv:2405.07798, 2024.
Angehi Xu, Ling-An Zeng, and Wei-Shi Zheng. Likert scoring with grade decoupling for long-term action assessment. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3222-3231, 2022a. doi: 10. 00.1099/CPPR32682222..00002
Dejing Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, Xiangnan He, and Yueting Zhuang. Video question answering via gradually refined attention over appearance and motion. In ACM Multimedia, 2017.
Jinglin Xu, Yongming Rao, Xumin Yu, Guangyi Chen, Jie Zhou, and Jiwen Lu. Finediving: fine- grained dataset for arocedure-aware action quality assessment. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2949-2955 2958, June 2022b.
Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: large video description dataset for bridging video and language. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5288-5288999 1888-5296,15 2016.
Qinghao Ye, Haiyang Xu, Jiabo Ye. Ming Yan, Anwen Hu, Haowei Liu, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou. mplug-ow12: Revolutionizing multi-modal large language model with modality collaboration, 2023.
Xumin Yu, Yongming Rao, Wenliang Zhao, Jiwen Lu, and Jie Zhou. Group-aware contrastive gression for action quality assessment. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 7919-7928, October 2021.
Eric Zelikman, Yuhuai WU, Jesse Mu, and Noah Goodman. STar: Bootstrapping reasoning with reasoning. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Ad- vances in Neural Information Processing Systems, 2022. URL /
Hang Zhang, Xin Li, and Lidong Bing. Video-Ilama An instruction- tuneed audio-visual language model for video understanding. arXiv preprint arXiv:2306.0220 02858, 2023a.
14

---

Published as a conference paper at ICLR 2025
Renrui Zhang, Jiaming Han, Chris Liu, Aojun Zhou, Pan Lu, Hongsheng Li, Peng Gao, and Yu Qiao. LLaMA-adapter: Efficient fine-tuning of large language models with zero-initialized attention. In The Twelfth International Conference Learning Representations, 2024. URL https:
7 /ooenreview. net. formm? d=dAui
Shiyi Zhang, Wenxun Dai, Sujia Wang, Xiangwei Shen, Jiwen Lu, Jie Zhou, and Yansong Tang. Logo: A long-form video dataset for group action quality assessment. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2405-2414, June 2023b.
Shiyi Zhang, Wenxun Dai, Sujia Wang, Xiangwei Shen, Jiwen Lu, Jie Zhou, and Yansong Tang. Logo: A long-form form-formmo video dataset for group action quality assessment. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2405-2414, June 2023c.
Ziqiang Zheng, Yiwei Chen, Jipeng Zhang, Tuan-Anh Vu, Huimin Zeng, Yue Him Wong Tim, and Sai-Kit Yeung. Exploring boundary of gpt-4v on marine analysis: A preliminary case study. arXiv preprint arXiv:2401.02147 2024.
Xingcheng Zhou and Alois C Knoll. Gpt-4v as traffic assistant: An in- look at vision language model on complex traffic events. arXiv preprint arXiv:2402.02205, 2024.
15


Reviews:
[{"review_id": "A8pJl6AdFC", "reviewer": "Reviewer_MNYz", "summary": "The paper introduces a novel self-training approach, Video Self-Training with augmented Reasoning (Video-STaR), to improve the performance of Large Multi-modal Models (LMMs) in video instruction tuning. By leveraging any labeled video dataset, Video-STaR enhances video question answering and adapts LMMs to new downstream tasks. The method cycles between generating answers, verifying them using the video labels, and fine-tuning the model. Experimental results show significant improvements in video understanding tasks, such as a 6.1% increase in Video QA performance and enhanced action quality assessment accuracy.", "questions": "1. **Clarification on Dataset Labeling**: Can the authors clarify how they handle datasets where video labels are ambiguous or not well-defined? It would be helpful to understand how such cases are managed during the label verification step to avoid generating incorrect instructions.\n\n2. **Self-Training Cycles**: Could the authors provide more details on how they determine when the self-training process has plateaued? Is there a specific performance threshold or metric used to stop the cycles?\n\n3. **Hallucination Mitigation**: The paper acknowledges hallucination as a potential issue, especially in label rationalization. Are there additional methods or heuristics that the authors could explore to reduce hallucinations during both answer generation and rationalization phases?\n\n4. **Generalizability to Complex Tasks**: Video-STaR shows notable improvements in performance on diverse video tasks. Could the authors elaborate on how the method handles particularly complex reasoning tasks beyond action recognition and quality assessment? How might this approach generalize to more abstract tasks, such as subjective video analysis?\n\n5. **Computational Overhead**: Given the iterative nature of the self-training process, what are the computational requirements and trade-offs for implementing Video-STaR at scale? Would this method be feasible for larger video datasets, and if so, what optimizations could be applied?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- **Originality**: Video-STaR introduces an innovative self-training approach for video instruction tuning, enabling the use of any labeled video dataset for instruction tuning. This represents a novel solution to a critical problem in video QA and reasoning by removing the dependency on manual dataset collection.\n\n- **Quality**: The paper is well-structured, with detailed experimental evaluations demonstrating significant improvements in video QA accuracy and adaptability to downstream tasks. It also addresses weaknesses in previous methods, such as reducing hallucinations in model answers.\n\n- **Clarity**: The methodology, including the self-training cycles of generation, filtering, and tuning, is clearly articulated, with effective visual aids to illustrate the process.\n\n- **Significance**: The paper contributes meaningfully to the field of video QA and LMM adaptation, showing improvements in both general video understanding and task-specific performance. It also creates a large and diverse dataset (VSTAR-1M), further enriching the resources available for multimodal research.", "weaknesses": "- **Computational Intensity**: The iterative nature of the self-training cycles, involving both answer generation and label rationalization, introduces significant computational overhead. This could limit the scalability and efficiency of the proposed method, especially in resource-constrained environments.\n\n- **Hallucination in Rationalization**: The reliance on label rationalization, especially for difficult tasks, may increase the likelihood of hallucinations. This undermines the robustness of the system in generating accurate explanations and answers, particularly in complex video tasks like FineDiving.\n\n- **Generalizability of Label Rationalization**: While the method assumes that all video labels require rationalization, certain straightforward labels may not benefit from this additional step. This leads to unnecessary computational load without proportional gains in performance.\n\n- **Limited Dataset Variety**: The study largely focuses on specific datasets like FineDiving, STAR-benchmark, and Kinetics700. Expanding the evaluation to a broader set of tasks could help demonstrate the full generalizability and adaptability of the approach to more diverse real-world video datasets.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8qTQ7VzeSD", "reviewer": "Reviewer_AVhL", "summary": "Large Multi-modal Models have achieved great success recently. However, when the problem comes to improve the reasoning capabilities of LMMs, the first and huge challenge is to get large scale and high quality datasets. Especially for video modalities, the problem is more hard by the difficulty and time-consuming. In this paper, the authors proposed Video-STaR, which is a method to generate video datasets with rationales based on LMMs. Besides, the authors releases a new video datasets named VSTAR-1M generated by the method.", "questions": "* The VSTAR-1M has about 1M videos mentioned in the paper but the experiments only tuned with 550k videos shown in Table 4. May I ask the authors if we apply the full VSTAR videos to tune LMMs, whether the results would be better?\n* Since neither the paper nor the appendix describes the verify process carefully, I briefly skimmed through the code and found that there is a 'verifier_type' for each sample. Is the type manually labeled for different samples or is it automatically judged by some rules?\n* In the paper it is mentioned that bounding boxes can be used as labeled data, so does this data enable LMMs to localize objects better than before? For example, to get better results when doing video segmentation or object tracing or simply image object detection?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "* The topic of automatically generating video datasets with accurate rationales is interesting and important for future multi-modality researches. It is very difficult to use human labor to fine-grain annotate videos with various reasoning details. \n* The authors release a large scale video instruction tuning dataset with rich CoT reasoning which significant saves the time and computing resources for future works.", "weaknesses": "* Limited technical novelty. The idea of this paper is very similar to STaR and all the proposed modules including answer generation, label rationalization and label verification are processed only for text modality, with no special considerations for video modality. It means that it is easy to transfer from normal STaR. Besides, for the methods section, the authors description is not clear and it is better to write it with more details\n* Lack of experimental description, theoretical analysis, or other deeply analysis of some assumptions. In the paper, the authors claim that Label Rationalizations have higher possibility to result in hallucination. However, the authors don't provide further analysis, nor do some experiments to prove it (even only on some partial samples and do significance testing). Rather, the authors propose a case to show the problem in Figure.3. However, in cycle 1 I think the problem lays in LMMs repeats the questions instead of hallucination, and in cylce 2, Answer Generation also get a non-perfect result \"with an overall score of 64.68\" as the ground truth is 65.6", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZLv0OmRCKA", "reviewer": "Reviewer_gCco", "summary": "This paper addresses the challenge of leveraging existing video datasets with diverse non-instruction labels to enhance video instruction tuning. The authors introduce Video-STaR, an automated loop for instruction generation, filtering, and self-tuning. Notably, they incorporate a Label Rationalization step within the loop to backtrace the rationale from video labels on challenging data. Experiments demonstrate that Video-STaR enables Large Multimodal Models to adapt to new tasks effectively.", "questions": "1. Recent studies indicate that using data generated by large language models (LLMs) for self-training can ultimately have a catastrophic impact on model performance. How do the authors address this issue? An interesting experiment could involve alternately training two models—Video-LLaVA (Model A) and another model (Model B)—using the Video-STaR strategy. In this setup, results generated by Model A could fine-tune Model B, and subsequently, results from Model B could fine-tune Model A. Observing the outcomes could reveal any differences in performance and stability under this iterative training approach.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. This paper addresses a significant issue: how to utilize existing video dataset labels to improve instruction tuning. This paper presents the first self-training pipeline that could inspire future approaches in this area\n2. The writing is clear. The authors also conduct extensive experiments to validate the proposed Video-STaR approach.", "weaknesses": "1. **Missing Experiments**: There is a lack of validation on the effectiveness of label filtering, including an ablation study on the verifier and clarification on why ground-truth (GT) labels should be included in the answers.\n2. **Heuristic and Inflexible Label Verification**: The label verification process is heuristic, limiting flexibility and scalability. Since different videos contain diverse labels, the proposed approach requires custom parsers and verifiers for varying label formats. For example, timestamps may need distinct regex patterns due to format variations. Additionally, label matching, such as “chopping wood” vs. “smashing,” as seen in Table 2, appears to demand considerable human effort.\n3. **Weak Baseline in Video-LLaVA Gemini**: Video-LLaVA Gemini is a relatively weak baseline. In Table 4, the authors used Gemini-pro to label only 2K samples for fine-tuning, a notably smaller dataset than those used for other baselines. The choice of 2K samples is not straightforward. The authors should attempt to label more videos with Gemini and include a curve showing the relationship between labeled data quantity and fine-tuning performance. How much labeled data is needed to achieve results comparable to Video-STaR’s 550K in both Table 3 and Table 4?\n4. **Marginal Cross-Dataset Generalization (Table 4 & Table 7)**: Video-STaR’s improvement in cross-dataset generalization appears marginal. The authors attribute this to noise within the benchmark itself. If this is the case, comparisons and ablation studies across these four datasets may not convey valuable information. The authors could consider using alternative high-quality datasets to better evaluate Video-STaR’s true effectiveness in enhancing cross-dataset generalization.\n5. **Typos**: Table index is incorrect. Line 260 Table 2.3 --> 2. Line 306 Table 3.2 --> 2", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7nuR8mR21Z", "reviewer": "Reviewer_ke7n", "summary": "The paper proposes a novel method to perform instruction tuning of Large Multi-modal Models (LMMs) on labeled video datasets, where the label format is arbitrary and different from that needed for instruction tuning. The main idea is to use the original LMM itself to generate question-answer pairs for instruction tuning, while using the original dataset labels as a \"verifier\" to filter out low-quality generations. The authors perform multiple cycles of dataset generation and model training to obtain the final model Video-STAR. Video-STAR shows improved performance on TempCompass and zero-shot video QA datasets.", "questions": "* In Tables 6 and 7, the last row \"- Generation\" means that neither generation nor rationalization are used, or only that generation is not used while rationalization is used? Either way, it is important to see both experiments to better understand the contribution of each method.\n* When doing self-training in multiple cycles, every time the model is initialized from the same non-instruction-finetuned checkpoint. Why not reuse the model from the previous cycle of self-training as initialization? \n* How do the authors deal with numerical labels such as temporal localization, bounding box, performance score? Do they expect the LVM to predict those directly and compute the L1 distance in the verifier? If so, how accurate is the prediction of the model in this case? Is Answer Generation capable of generating good question-answer pairs or Label Rationalization is more helpful here?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "* The idea of turning existing labeled video datasets into instruction tuning dataset and use them for self-training is elegant and effective.\n* The paper is well written and is easy to follow\n* I appreciate the ablation study", "weaknesses": "* The main weakness of the paper is the Video-LLaVA baselines, i.e. Video-LLaVA+ and Video-LLaVA-gemini, which are the main points of comparison in the paper. Video-LLaVA-gemini has been fine-tuned only on several thousand sample pairs, which is incomparable to hundreds of thousands of video samples used for Video-STAR. As for Video-LLaVA+, the number of training samples is comparable, yet there is no information about how those samples were obtained. The way the samples in Video-LLaVA+ are constructed from the original video labels would affect the final model performance a lot, so it is important to explain this in details.\n* Additionally, one important baseline for dataset creation is missing. It looks like the SoTA method for creating instruction datasets from video is VideoInstruct [A]. I would expect the authors to compare to this method of dataset creation, both in answer generation and label rationalization settings, to understand how much the video stream actually helps.\n* It is somewhat unclear from the paper if Label Rationalization can be used to generate the entire dataset, without using Answer Generation. Somewhere in the paper the authors suggest that it leads to hallucinations, yet in Tables 6 and 7 they show the improvement.  \n\nThis is a preliminary rating and I will revise my score once the weaknesses and questions are addressed.\n\n[A] -  Maaz et al. \"Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Models\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)