VideoCoT: A Video Chain-of-Thought Dataset with Active Annotation Tool (2024)

Abstract: Multimodal large language models (MLLMs) are flourishing, but mainly focus on images with less attention than videos, especially in sub-fields such as prompt engineering, video chain-of-thought (CoT), and instruction tuning on videos. Therefore, we try to explore the collection of CoT datasets in videos to lead to video OpenQA and improve the reasoning ability of MLLMs. Unfortunately, making such video CoT datasets is not an easy task. Given that human annotation is too cumbersome and expensive, while machine-generated is not reliable due to the hallucination issue, we develop an automatic annotation tool that combines machine and human experts, under the active learning paradigm. Active learning is an interactive strategy between the model and human experts, in this way, the workload of human labeling can be reduced and the quality of the dataset can be guaranteed. With the help of the automatic annotation tool, we strive to contribute three datasets, namely VideoCoT, TopicQA, TopicCoT. Furthermore, we propose a simple but effective benchmark based on the collected datasets, which exploits CoT to maximize the complex reasoning capabilities of MLLMs. Extensive experiments demonstrate the effectiveness our solution.

heading: 1 Introduction
text: With the emergence of ChatGPT1, large language models (LLMs) have experienced unprecedented growth and have gradually expanded into the multimodal domain. Pioneers have explored multiple feasible paths around multimodal large models (MLLMs), such as training MLLMs from scratch (e.g. Kosmos-1 (Huang et al., 2023)), or bridging LLMs and vision modules (e.g. BLIP-2 (Li et al., 2023b)). Moreover, prompt engineering, chain-
*Corresponding author. 1https://openai.com/blog/chatgpt
Question: Why does [person_1] have to stop?
A: Because it is controlled by [person_1]. B: Because [person_1] can't eat anymore. C: [person_1] ran into a newly - spread bitumen road. D: Because [person_1] has to use the machine to do the exercise. E: Because [person_1] is hurdler.
Video: 3IAUsdx5C8_000004_000014
(a) Spatio-temporal changes in video.
(b) Significant differences among options.
Question: Why does [person_1] have to stop? A: Because it is controlled by [person_1]. B: Because [person_1] can't eat anymore. C: [person_1] ran into a newly - spread bitumen road. D: Because [person_1] has to use the machine to do the exercise. E: Because [person_1] is hurdler.
of-thought (CoT), and instruction tuning for multimodal LLMs are also flourishing. However, the majority of current research focuses on images, with video research (Deng et al., 2022; Zeng et al., 2022) remaining underdeveloped. For instance, Alayrac et al. (2022) employs a video understanding model to extract features, which are then inputted, while Ye et al. (2023) utilizes multiple frames of the video as input. Similarly, few researchers have devoted attention to sub-fields such as video prompt engineering (Li et al., 2023a; Zeng, 2022), and video instruction fine-tuning (Zhang et al., 2023b). We attribute this phenomenon to the fact that MLLMs are less mature than LLMs that solely rely on natural language input, and there are still numerous issues to be explored.
To advance the development of MLLMs for videos, our primary interest lies in CoT in videos. Video CoT has multiple benefits as follows: 1) Towards OpenQA in video. Currently, the VideoQA dataset widely adopts the form of multiple-choice questions, but there are significant differences between the answer options (Kamalloo et al., 2023). As illustrated in Fig.1(a), the options between A-E are significantly different, especially the descriptions of eating and being a hurdler are completely irrelevant to the video. This fact lead to models
92
finding shortcuts to the dataset pattern. 2) Enhance understanding. Videos contain more temporal and spatial changes than images, and CoT can help capture the complex semantics of these changes (Zeng et al., 2021). As shown in Fig.1(b), the key to solving the question, that is, the girl changes from moving to stopping (temporal) and the appearance of the bitumen road (spatial), is to develop with the video. 3) Improving the reasoning ability of MLLMs. A more logical CoT can enhance the reasoning ability of MLLMs when used for training.
Although video CoT shows great potential, creating a video CoT dataset is a non-trivial task. The process of fully annotating CoTs by humans is both tedious and expensive, which is why we aim to develop an automatic pipeline for generating CoTs. Intuitively, one widely adopted strategy is to use off-the-shelf MLLMs or LLMs as assistants for reasoning. However, there are several challenges that need to be addressed. Firstly, MLLMs do not possess strong reasoning abilities and cannot directly generate reliable CoTs. Secondly, while LLMs have reasoning capabilities, they cannot use images as input for CoT generation. Lastly, machinegenerated data is often unreliable due to ethical doubts and hallucination issues (Liu et al., 2023; Qin et al., 2023), which require human correction for quality control.
Therefore, in this paper, we develop an automatic annotation tool that combines machine and human experts, under the active learning paradigm (Zhang et al., 2023a). As shown in Fig.2, active learning is a strategy that involves interaction between the model and human experts, where the model actively seeks the opinions and standards of experts when encountering difficult samples (Zhai et al., 2022). In this way, the workload of human la-
beling can be reduced and the quality of the dataset can be guaranteed in the process(Wu et al., 2024; Lu et al., 2021). Specifically, we will train a prompt generator to guide LLMs to generate complex CoT based on video information. Meanwhile, we will formulate a quality score to evaluate the generated CoT sentences from multiple aspects. Among them, low-quality sentences will be modified by human experts, and the modified CoT will be used to train the prompt generator to guide LLMs to generate more reasonable CoT(Guo et al., 2022; Liu et al., 2022).
With the help of the aforementioned automatic annotation tools under the active learning paradigm, we strive to contribute three videoCoT datasets, namely VideoCoT, TopicQA, TopicCoT. Among them, VideoCoT is designed to supplement CoT between question and answer from existing datasets. Furthermore, we leverage the topic items in the dataset to construct TopicQA, which enables MLLMs to learn the relevant relationship between videos and topics, and TopicCoT, which facilitates reasoning about the topic relevance. Furthermore, we apply these datasets to propose a simple benchmark. Extensive experiments demonstrate the effectiveness of our datasets and solution. The main contributions are summarized as follows:
• To the best of our knowledge, this is the first work that introduces an automatic annotation tool under the active learning paradigm for complex CoT generation in the video domain.
• We have collected three dataset to fill the vacuum of Video CoT via our automatic annotation tool, namely VideoCoT, TopicQA, TopicCoT.
• We propose a simple but effective benchmark
based on the collected datasets, which exploits CoT to achieve better reasoning ability.

heading: 2 Related Work

heading: 2.1 Multimodal Large Models
text: As a result of the flourishing development of LLMs (Pan and Zeng, 2023), many frameworks and techniques have been extended, such as prompt engineering, chain-of-thought, and instruction tuning. In the field of multimedia, these hotspots are still the topic of discussion (Li et al., 2024). Subsequently, Zhu et al. (2023) proposed mini-GPT4, Li et al. (2023b) introduced blip2, and Ye et al. (2023) intruduced mPLUG-OWL. However, the majority of current research focuses on images, with video research remaining underdeveloped. To fill the academic vacuum, we propose an automatic annotation tool under the active learning paradigm, and further collect three datasets based on it. In this way, the complex reasoning ability of MLLMs is improved (Rajesh et al., 2023; Zeng et al., 2024).

heading: 2.2 Chain-of-Thought
text: Chain-of-Thought (CoT) has been proven to be an effective strategy to enhance reasoning, and its effectiveness has been widely demonstrated in the field of LLMs (Ma et al., 2023). In the field of multimedia, works such as ScienceQA (Lu et al., 2022) and VisualCoT (Rose et al., 2023) have also been proposed. Inspired by the above work, we try to extend the potential of CoT in the field of video understanding, which helps improve the reasoning ability of MLLMs.

heading: 3 Dataset Collection
text: Following Causal-VidQA (Li et al., 2022), we built three datasets around videos based on Kinetics700, namely VideoCoT, TopicQA, and TopicCoT. In this section, we will introduce the process of active annotation tool, on which both VideoCoT and TopicCoT are collected.

heading: 3.1 Active Annotation Tool
text: Fig.2 illustrates the pipeline of our automatic dataset construction approach, which implements the prompt generation for LLMs under the active learning paradigm to generate the logical CoT processes. Active learning is an interrogation method between the model and human experts (Zhang et al., 2023a), which reduces the annotation workload and guarantees the quality of the dataset.
Specifically, the automated process is divided into three steps, namely prompt generation, automatic scoring, and expert refinement. Among them, prompt generation aims to generate suitable prompt to guide LLMs to generate comprehensive and reasonable CoT, while automatic scoring checks the quality of machine-generated CoT from multiple quality dimensions. Among them, the low-quality CoT will be refined and modified by experts, which is also used to train the prompt generator to improve the quality of CoT generation.

heading: 3.1.1 Prompt Generation
text: We try to drive the off-the-shelf LLMs (i.e. GPT-4) to generate some high-quality CoT data for us, but unfortunately, the logic of the generated sentences obtained by the fixed template (i.e. prompt) is incomplete and incoherent. Therefore, we introduce a prompt generator to maximize the potential of
guiding LLMs and ultimately reduce manual labor. Specifically, we borrow a summarization model (Rao et al., 2021) capable of handling long sentences as the prompt generator, which will be trained in interaction with human experts. In the initial stage, it is fed a long video description, a question and a answer, and finally outputs a short summary. Obviously, such a prompt is difficult to guide LLMs to get a reasonable CoT between the question and the answer, so it needs to learn from human modified sentences. We will present the scoring mechanism and human refinement in the next subsection.
After multiple rounds of iterations, the generator will flexibly deal with different videos to generate corresponding prompts. Thereafter, since MLLMs do not yet have good reasoning capabilities (which is what we hope to do), we still implement generation based on LLMs (i.e. GPT-4). Finally, after manual inspection with less labor, a reasonable CoT can be obtained, as shown in the Fig.3.

heading: 3.1.2 Automatic Scoring
text: In order for a quality-required CoT to be generated, we believe that a high-quality CoT CvCoT 2 should have both: 1) the generated sentences are fluent, 2) a comprehensive understanding of objects and relations, 3) and reasonable reasoning between the question and the answer. To achieve this, we design a scoring function SvCoT that automatically evaluates from six dimensions, i.e., perplexity Sppl, background Sbac, temporal changes Stem, spatial objects Sspa, relations Srel, summary Ssum.
SvCoT = Sppl+Sbac+Stem+Sspa+Srel+Ssum. (1) Among them, the “perplexity” evaluates the fluency of generated CoT, and its reciprocal is used as part of the quality score (Basu et al., 2021). This score is closer to 1 when the CoT sentence CvCoT is more fluent.
Sppl = 1
PPL(CvCoT ) . (2)
The “background” Sbac indicates whether the generated CoT describes the video scene or not. We collect some keywords to evaluate this, i.e., when a sentence of CoT has words such as background, video scene, etc., it is considered to meet the qual-
2Cv represents “video”, while CvCoT represents VideoCoT, which serves to differentiate it from TopicCoT CtCoT .
ity requirement. Sbac = {
1 if the video scene is described in CvCoT 0 otherwise
(3) The “spatial objects” Sspa and “temporal changes” Stem represent how many objects and actions are included in the generated CoT, respectively. The objects and actions (extracted by GRiT(Wu et al., 2022)) that should be included are taken as the evaluation criteria, i.e. the more objects and actions are included in CvCoT , the higher the score Sspa and Stem. Conversely, if irrelevant objects or actions appear in the sentence CvCoT (most likely hallucinations), the score will be negative.
Sspa = poso(CvCoT )− nego(CvCoT )
ground_truth(CvCoT ) , (4)
Stem = posa(CvCoT )− nega(CvCoT )
ground_truth(CvCoT ) , (5)
where poso and posa indicate the number of objects and actions present in the CoT, where pos indicates real presence in the video, and neg indicates hallucinated objects or actions.
The “relations” Srel represents whether the generated CoT has the analysis of spatio-temporal relationship among objects, and the connection with video scene. And the “summary” Ssum evaluates whether a summary is included in the generated CvCoT (i.e., the answer is output via step-by-step
reasoning). Srel = {
1 if the analysis is included in CvCoT 0 otherwise
(6)
Ssum = {
1 if the summary is included in CvCoT 0 otherwise
(7) All the above scores belong to the interval from 0 to 1, which is convenient for us to do further normalization. The automatic score S serves as a “rough indicator" to identify the worst sample and help us optimize prompt generator. In particular, since Sspa and Stem are more important for this task, we set the balance parameters in Eqn.1 as (0.1, 0.1, 0.3, 0.3, 0.1, 0.1). Furthermore, to control the quality of CoT, when the normalized score is lower than 0.9, it will be sent to human experts for refinement.

heading: 3.1.3 Expert Refinement
text: We enlisted ten human experts with backgrounds in artificial intelligence to participate in the annotation process. To ensure consistency in the labeling results across different experts, a 5-rounds pre-annotation training was conducted prior to official annotation. Specifically, each expert was required to label a small number of samples to gain an understanding of the annotation rules, which were standardized to ensure consistency among all participants.
For the generated CoT whose quality score is less than the threshold (i.e. 0.9), they will be modified by human experts. As much as possible, experts are asked to make sentences include scene descriptions in video, spatio-temporal relationships,
and logical reasoning between the question and answer. Meanwhile, the refined samples will return to the dataset pool and participate in training of prompt generation until the quality of all annotations meets our requirements. Through this interactive active learning paradigm, the high-quality CoT are semi-automatically constructed.

heading: 3.2 Automatic Datatset Construction
text: With the help of the aforementioned annotation tool under the active learning paradigm, we strive to contribute three datasets, namely VideoCoT, TopicQA, TopicCoT.

heading: 3.2.1 VideoCoT
text: VideoCoT is designed to supplement CoT between question and answer from existing datasets, CausalVidQA. Based on the settings, we collect 11, 182 samples containing CoT, as shown in Table 1.

heading: 3.2.2 TopicQA
text: Further, we leverage the topic items in the Kinetics700 dataset to construct TopicQA, which enables MLLMs to learn the relevant relationship between videos and topics. In this dataset, we take “is the video relevant to the topic” as the question and “yes” or “no” as the answer.

heading: 3.2.3 TopicCoT
text: TopicCoT, similar to the construction process of VideoCoT, which contains step-by-step reasoning between questions and answers in TpoicQA. Specifically, TopicCoT CtCoT is still based on our automatic annotation tool, but the scoring function is different, which is defined as follows:
StCoT = Sppl +Stem+Sspa+Scon+Ssum. (8)
where Scon represents the concept of the topic, and the others are consistent with Eqn.1. Moreover, the balance parameters are set to (0.1, 0.2, 0.2, 0.4, 0.1) for normalization. Then, when this score StCoT is less than 0.9, it will be sent to humans for modification.

heading: 3.3 Dataset Analysis

heading: 3.3.1 Property Quality
text: The statistical analysis of textual description in our VideoCoT and TopicCoT dataset is shown in Fig.5. Based on statistical results, the original dataset, which includes both questions and multiple choices, has an average length of approximately 50 words. In contrast, the rationale length of our VideoCoT and TopicCoT is distributed between 100 and 150 words.

heading: 3.3.2 Diversity Quality
text: To assess the diversity of sentences in the VideoCoT and TopicCoT datasets, we conduct a word frequency analysis of nouns, verbs, and conjunctions, which represent descriptive, temporal, and logical aspects, respectively. Fig.6 illustrates the top 5 frequency of each category in the rationale of the two datasets. 1) Noun: We observe that the high-frequency nouns in VideoCoT mostly refer to specific objects, such as “person" and “man", as well as key words in the reasoning process, such as “scene", “answer" and “function". In contrast, the top nouns in TopicCoT mainly involve “topic" and “concept", indicating that detailed descriptions revolve around the topic and object concepts of the video. 2) Verb: The main verbs in VideoCoT describe specific human activities, focusing on the temporal aspect of the video. In TopicCoT, the
high-frequency verbs are mostly reasoning verbs, focusing on the association between the question and the topic of the video. 3) Conjunction: The conjunction with the highest frequency in both datasets is “therefore", which indicates the logical and summary aspects of the rationale.

heading: 3.3.3 Visualization Quality
text: To verify the rationality of the human experts’ operation, we also check some cases as shown in the Fig.3. There are two languages present in our dataset, namely English and Chinese. The initial generated by LLMs was of low quality, which hindered the establishment of relationships. However, after undergoing multiple round of interaction between human and model, the score of generated CoT increased from 0.07 to 0.97 points, indicating a significant improvement in the quality of the output.

heading: 4 Proposed Method
text: The overall training framework is depicted with an illustration in Fig.7. For the task of video question answering (Zhong et al., 2022), multiple choice (MC) is more popular, but the differences between the options are too significant, and it is easy for the model to find shortcuts. Therefore, we are committed to achieving a free-form open-ended (OE) with logic rationale (Lu et al., 2022).

heading: 4.1 Training strategy of original dataset
text: The input of MC strategy is defined as X= (XQ, XMC , XV ), where XQ represents the question, XMC represents answer options, and XV represents the image.
Following the work of (Kamalloo et al., 2023), who trains the model using fixed long sentence
templates with correct options for filling in the blanks, the probability of generating an answer can be formulated as follows:
p(Y |XQ, XMC , XV ) (9)
= m∑
t=1
log p(yt|y<t, XQ, XMC , XV ), (10)
where Y = (y1, y2, . . . , ym) represents the target tokens.

heading: 4.2 Training strategy of VideoCoT
text: Similarly, the input of OE strategy is defined as X= (XQ, XV ).
In this way, the input X will be removed the answer options XMC , while the target answer Y will be redefined as the rationale R= (r1, r2, . . . , rn).
Formally, the probability of generating rationale can be formulated as follows:
p(R|XQ, XV )= n∑
t=1
log p(rt|r<t, XQ, XV ). (11)
Through this training strategy of CoT, more prior knowledge of MLLMs can be invoked, and finally answer questions through logical reasoning.

heading: 5 Experiments

heading: 5.1 Experimental Settings

heading: 5.1.1 Datasets
text: Our datasets are split into 3 non-overlapping subsets, where 0.6, 0.2 and 0.2 are used for training, validation and testing.

heading: 5.1.2 Evaluation Protocol
text: We adopt accuracy as our evaluation metric, which is utilized to measure whether the answers generated by models are correct. Notably, in the
multi-choice setting, the accuracy AccMC can be directly compared with ground-truth. In the case of open-ended QA, we adopt two metrics, 1) AccOE(keywords): whether the “summary" sentence hits the keywords in the ground-truth answer. Specifically, keywords and their synonyms are acquired by giving some few-shot template and QA pair to GPT4. We then calculate the correct proportion of keywords for each question as its score. 2) AccOE(GPT-4): regard GPT-43 as a referee to evaluate semantic relevance.

heading: 5.1.3 Baselines
text: We select the following models as our baselines: mPLUG-Owl (Ye et al., 2023), VisualGLM (Du et al., 2022), mini-GPT4 (Zhu et al., 2023).

heading: 5.2 Overall Performance Comparison
text: To verify the effectiveness of our datasets, we train several MLLMs with the original dataset and our datasets respectively4. Among them, for the evaluation of OE task, we adopt two kinds of metrics, namely a hard metric (based on keywords) and a soft metric (based on GPT-4).
The experimental results are presented in Table 2, and the following observations can be made: 1) In comparison to the multi-choice setup, both models exhibit improved performance in open-ended QA accuracy. Upon analyzing the multi-choice outputs, it is evident that the models often provide justifications for each individual option rather than selecting a single response to address the given question. 2) The superiority of both VideoCoT trained MLLMs over the original method is evident in the improvements observed across both keyword
3https://openai.com/product/gpt-4 4TopicQA is an ordinary QA dataset, which will not be adopted to discuss the impact of CoT on reasoning ability, but it can still be a traditional QA dataset.
and GPT-4 metrics. This highlights the significant impact of employing a chain of thoughts within the generation model’s creative process. 3) We also observe that the accuracy of keywords on all models surpasses the accuracy of GPT-4, which is due to the former metric being more relaxed than the latter. 4) Additionally, we conduct an experiment utilizing a hybrid training dataset comprising both VideoCoT and TopicCoT. The subsequent evaluation of models take place on the testing of VideoCoT. Remarkably, when contrasted with models solely trained on VideoCoT, the GPT-4 metric exhibited a noteworthy improvement through hybrid training. This improvement surpassed the performance of all models that are only trained on VideoCoT. This outcome serves as a compelling indicator that hybrid training fosters a reciprocal influence, allowing models to acquire the capacity for incremental and reasoned thinking.

heading: 5.3 Reasoning Ability Visualization
text: The visualization is shown in Fig.8, the mPLUGOwl possesses the capability to depict the content of the image and execute the basic task of question and answer. However, its performance is unsatisfactory when confronted with more complex questions that necessitate reasoning. Conversely, upon being trained on our datasets, it acquires the ability to identify objects in the image (e.g. “a group of people”), discern the fundamental functions of objects or events (e.g. “the essential fuction of”), and finally integrate objects and relationships to engage in reasoning (e.g. “because they might participating in a fitness event”).

heading: 6 Conclusions
text: In this work, we strive to explore the collection of CoT datasets on videos to bootstrap OpenQA on videos and improve the inference ability of MLLMs. To reduce the cost of manual annotation, we develop an automatic annotation tool that com-
Round 1 An apple is peeling by a peeler and the peeler is pressed. They are playing the apple and peeler. (一个苹果正在用削皮器削皮，削皮器被压着。他们在玩苹果削皮游戏)
Round N First, the scene takes place in a toy store where a person is peeling an apple using a hand-held peeler. They are holding the apple in their hand and using the peeler to remove the skin. (首先，场景发生在一家玩具店，一个人正在用手持削皮器剥苹果。他 们手里拿着苹果，用削皮器去皮)
The essential function of the handheld apple peeler is to remove the skin from the apple. It is a small tool that can easily slip or move around while in use, so it needs to be held stable to ensure efficient and effective peeling. This action allows them t control the depth and angle of the peeler, ensuring that they remove only the skin and not too much of the flesh. (手持式苹果削皮机的基本功能是去除苹果的果皮。它是一种小型工具， 在使用时可以轻松滑动或移动，因此需要保持稳定，以确保高效和有 效的剥离。这个动作允许他们控制削皮器的深度和角度，确保他们只 去除皮肤，而不是太多的果肉)
Therefore, the answer is that the person presses the handheld apple peeler to keep it stable. (因此，答案是，这个人按下手持苹果削皮器以保持其稳定)
Background
Spatial objects
Relations Summary
Temporal Changes
Perplexity
Video Score
bines machine and human experts, under the active learning paradigm. With the help of this annotation tool, we contribute three videoCoT datasets, namely VideoCoT, TopicQA, TopicCoT. Experimental results show that our datasets achieve superior effectiveness, diversity and explainability.

heading: Acknowledgements
text: This work is supported in part by the National Natural Science Foundation of China (62372187), in part by the National Key Research and Development Program of China (2022YFC3601005) and in part by the Guangdong Provincial Key Laboratory of Human Digital Twin (2022B1212010004).
Limitations
In regards to the active annotation tool, using our tool on additional datasets can enhance the visual reasoning abilities of more models. However, funding constraints limited the invitation of annotation experts. Nonetheless, we are committed to expanding the impact of this paper in future research. Moreover, our training resources currently restrict the application of our dataset to significantly more larger models.

heading: Future Work
text: Nonetheless, we are committed to expanding the impact of this paper in future research. Moreover, our training resources currently restrict the application of our dataset to significantly more larger models