Learnability Matters: Active Learning for Video Captioning

Authors: yiqian  zhang
buyu  liu
jun  bao
qiang  huang
min  zhang
jun  yu
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-13 07:14:19

Abstract: This work focuses on the active learning in video captioning. In particular, we propose to address the learnability problem in active learning, which has been brought up by collective outliers in video captioning and neglected in the literature. To start with, we conduct a comprehensive study of collective outliers, exploring their hard-to-learn property and concluding that ground truth inconsistency is one of the main causes. Motivated by this, we design a novel active learning algorithm that takes three complementary aspects, namely learnability, diversity, and uncertainty, into account. Ideally, learnability is reflected by ground truth consistency. Under the active learning scenario where ground truths are not available until human involvement, we measure the consistency on estimated ground truths, where predictions from off-the-shelf models are utilized as approximations to ground truths. These predictions are further used to estimate sample frequency and reliability, evincing the diversity and uncertainty respectively. With the help of our novel caption-wise active learning protocol, our algorithm is capable of leveraging knowledge from humans in a more effective yet intellectual manner. Results on publicly available video captioning datasets with diverse video captioning models demonstrate that our algorithm outperforms SOTA active learning methods by a large margin, e.g. we achieve about 103% of full performance on CIDEr with 25% of human annotations on MSR-VTT.

Main paper: 
 introduction: 
 Video captioning, which aims to understand videos in the form of describing them in natural language Abdar et al. (2023) , becomes a heated task Lin et al. (2022) with the emergence of a large amount of data Li et al. (2023a) as well as transformer-based models Liu et al. (2022) . Despite the superior performance, existing methods suffer heavily from the need for time-consuming and labor-intensive human annotations Chan et al. (2020) because of their learning-based nature.Approaches such as active learning Tharwat and Schenck (2023) , semi-supervised learning Yang et al. (2023) , and domain adaptation Yu et al. (2023) are proposed to address the above-mentioned data issue. This paper takes the active learning approach where we assume there exists a small amount of labelled data together with a large number of unlabelled ones. Our goal is to select the most informative samples from the unlabelled set and have them annotated by humans such that the video captioning model can achieve the best performance with the minimum human effort. Though uncertainty and diversity have been exploited to investigate the properties of unlabelled data, theAbstraction Inconsistency (a)• a boy and girl are sitting by each other while he drinks tea.• a commercial about a beverage. (High-level semantics)• a boy is taking a surf board to the ocean.• the child on the surfboard becomes the grown man in the movie clip. (Reasoning)• a cartoon character talking on the phone.• Krabs talks on the phone while SpongeBob freaks out. (External knowledge)• a map with many red arrows showing how different events took place there. • man narrating while a map of Europe is displayed talking about the industrial revolution that started in England. (OCR)a woman is singing a person in a kitchen is mixing a bowl of potato salad with a spatula in a white bowl on a cutting board a group of people perform on screen together potential problem from human annotations is largely neglected in active learning literature as these annotations are not available during the selection process.In this paper, we first and foremost propose to analyze problems of human annotations in video captioning tasks (see Fig. 1.(b) ). With the help of Dataset Maps, we are able to observe the problem, or collective outliers, in human annotations, and conclude its severity by the amount of outliers. Such an amount will further aggravate the learning process due to their hard-to-learn nature, thus worth exploring. We observe that inconsistency in human annotations is one of the main causes of collective outliers, which can be divided into the abstraction and granularity inconsistency (refer to Fig. 1 .(a)).The former highlights instances where humans offer different abstractions for videos with complex contents, while the latter reveals that humans may describe the same object at varying semantic levels.To this end, we incorporate our observations into active learning frameworks. We re-phrase the problem introduced by collective outliers as learnability and aim to address it in our active learning design. Due to the lack of access to human annotations in active learning, directly identifying collective outliers is implausible. Nevertheless, we turn to their main property, or inconsistency, as our breakthrough and propose to estimate the abstraction and granularity inconsistencies in unlabelled data. Concretely, existing image-based large Vision-Language Models (LVLMs) are used to approximate human annotations. Instead of directly using predictions from LVLMs as pseudo ground truths, which would deteriorate the overall performance because of domain gaps, we propose to measure the consistency not only internally among per-frame predictions but externally between predictions from a video captioning model and those per-frame predictions. The internal measurement captures the abstraction consistency and the external one counts on the granularity consistency. Combining both provides us an estimation of sample learnability, or how likely this sample belongs to collective outliers. Diversity and uncertainty are also leveraged in our active learning scheme where we rank samples based on their frequency in the entire dataset and reliability respectively. Our algorithm is able to select reliable, diverse yet learnable samples, and achieves SOTA trade-offs between accuracy and human efforts. Motivated by the causes of collective outliers, we propose a caption-wise active learning protocol such that only a limited number of human-annotated captions are acquired if one video is selected by our active learning algorithm. Our protocol is capable of avoiding inconsistency and providing a more intellectual way to allocate human effort to more diverse videos. In all, our contribution can be summarized as follows:• To the best of our knowledge, we are the very first in terms of exploring collective outliers on video captioning tasks and providing comprehensive studies on them. • A novel active learning algorithm that explicitly takes learnability, diversity, and uncertainty into account. Specifically, the learnability is designed to tackle collective outliers, inspired by their abstraction and granularity inconsistency. • A novel caption-wise active learning protocol that effectively leverages knowledge from humans.• State-of-the-art performances on video captioning datasets under diverse model backbones.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes] Justification: The abstract and introduction accurately introduce the proposed active learning method.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
related work: 
 Visual Captioning Based on Deep Learning Visual captioning Sharma et al. (2023) is a task that automatically generates textual grammatical and semantically appropriate descriptions for a given visual content. While this task is easy for humans, it is extremely difficult for deep learning models. To perform visual captioning, models are required to have multiple capabilities: including but not limited to detection and recognition capabilities to extract objects in visual content, visual semantic understanding capabilities to determine the attributes and relationships of objects, and enriching language knowledge to describe information with grammatically correct sentences. Visual captioning can be classified into image captioning Ming et al. (2022) and video captioning Abdar et al. (2023) . Deep-learning-based image captioning starts with the encoder-decoder framework combining convolutional neural networks and recurrent neural networks Kiros et al. (2014) ; Vinyals et al. (2015) . It is further developed by extracting fine object region features Anderson et al. (2018) ; Karpathy and Fei-Fei (2015) , attribute features Yao et al. (2017) , semantic relation features Yao et al. (2018) ; Yang et al. (2022) and introducing attention mechanisms Xu et al. (2015) ; Pan et al. (2020) ; Yu et al. (2020) . With breakthroughs in vision-and-language pre-training approaches Radford et al. (2021) ; Zhang et al. (2021a) ; Alayrac et al. (2022) ; Li et al. (2023a) , applications in image captioning Li et al. (2020) ; Mokady et al. (2021) ; Li et al. (2023c) have emerged in recent years, benefit from the power of LVLMs (e.g.rich knowledge and strong recognition ability). The recently proposed BLIP2 Li et al. (2023a) has achieved superior performance on major image-text tasks with a simple pre-training method. Unfortunately, LVLMs focusing on video-text-related tasks have not yet come out. Video captioning still faces many challenges. Video captioning is more difficult than image captioning due to the many frames the video contains, which carry massive amounts of information. This raises the necessity of abstracting semantic information from the temporal dimension and describing content from the spatial dimension with different granularity. Video captioning started with the encoder-decoder framework Venugopalan et al. (2015) . Subsequent researchers continued to design new encoders Wang et al. (2019) ; Pan et al. (2016) 2021). Among them, Lin et al.proposed the end-to-end video captioning method Lin et al. (2022) for the first time and achieved significant performance improvement. Unlike existing video captioning methods, we focus on underlying data issues. Unlike existing video captioning methods, our work focuses on underlying data issues. We are committed to exploiting knowledge to measure video consistency scores for active learning.Active Learning Due to the high cost of manually annotating data, active learning Tharwat and Schenck (2023) has attracted widespread attention from academia and industry. Active learning aims to use as few, high-quality samples as possible to achieve the best possible performance of the model. Active learning has many mature works in closed tasks such as image classification Parvaneh et al. (2022) , object detection Wu et al. (2022) , semantic segmentation Xie et al. (2022) , and natural language processing Zhang et al. (2022) . Among them, Coreset Sener and Savarese (2018) can minimize the distance between an example in the unlabeled pool to its closest labeled example, and can effectively capture the diversity in the dataset. However, active learning for video captioning remains to be explored. Chan et al. Chan et al. (2020) tries to migrate conventional active learning methods to video captioning, and has proposed a new method based on ensemble and clustering. Besides, active learning has been observed to fail on open-ended tasks such as visual question answering due to the presence of collective outliers Karamcheti et al. (2021) . There is currently no literature that attempts to address the impact of collective outliers. In this work, we focus on the problem of collective outliers in video captioning. We point out possible causes of collective outliers and attempt to mitigate their impact on active learning methods.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [No] Justification: The limitations of this article may come from the capabilities of LVLMs.Experiments have proven that the capabilities of LVLMs can initially meet the needs.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
method: 
 As the very first work that explores data learnability in active learning for video captioning, we start our method with a comprehensive analysis of their causes and various forms in Sec. 3.1. According to our analysis, we then propose a novel active learning scheme that in particular considers the learnability during the unlabelled data selection process in Sec. 3.2, together with novel designs of uncertainty and diversity. Our overall active learning method can be found in Fig. 2 .Motivation The order of training samples is shown to be important in curriculum learning Matiisen et al. (2020) . Specifically, it found that feeding models with examples of successively increasing difficulty produces better performances than providing full examples immediately. Though posing a more severe problem in active learning as selecting far more difficult samples at early stages is a waste of human efforts, measuring sample difficulty is largely neglected in the literature as neither ground truth nor the definition itself can be easily obtained. To this end, our goal is to first identify difficult examples in video captioning task, and then perform analysis on them so that our observations can be beneficial for our active learning selection scheme.Methodology Collective outliers are a group of data objects that fall extremely far from well-defined norms of a data set or given concepts of expected behavior in data-mining Lai et al. (2021) , which are much harder to detect as they are more generic yet harder to identify as individuals. Not surprisingly, one important property of collective outliers is learnability, e.g., they are more likely to be hard-tolearn samples compared to normal data Karamcheti et al. (2021) . Concretely, we exploit Dataset Maps to diagnose datasets with training dynamics Swayamdipta et al. (2020) ; Dagan et al. (2013) ; Sakaguchi et al. (2020) . Compared to conventional Dataset Map that exploits two model-specific metrics (i.e.average confidence assigned to the correct answer and the variance of these values) to measure the learnability of training examples, we propose to utilize SPICE Anderson et al. (2016) to approximate the confidence score on the correct answer. On the one hand, the confidence score is not a reliable criterion as it can be largely affected by random-length text sequences generated by autoregressive models Zhang et al. (2020a) . On the other hand, SPICE measures the F1 score between scene graphs generated by predictions and ground truths. It has the useful property of being defined over tuples that are easy to subdivide into meaningful categories, providing a more semantic-centric evaluation in a human-interpretable manner by abstracting away most of the lexical and syntactic idiosyncrasies of natural language compared to CIDEr Vedantam et al. (2015) . Observation We provide an example of Dataset Map of SwinBERT Lin et al. (2022) trained on MSR- VTT Xu et al. (2016) in Fig. 1.(b) , where the y-axis and x-axis plots the average SPICE score over training epochs and their variability respectively. Clearly, samples that fall into the upper left area of this figure are easy-to-learn ones whose SPICE scores are consistently high. While the lower left area is occupied with hard-to-learn samples where no observable improvements can be found with longer training time. Unlike conventional close-end tasks where only a small proportion of samples are hard to learn, we observe that a noticeable amount of samples fall into the lower left area.Analysis Taking a closer look at collective outliers, we observe that the video captioning model struggles when learning from samples with inconsistent human annotations. As shown in Fig. 1 .(a), these inconsistencies can be divided into two categories, abstraction and granularity. The inconsistency in abstraction summarizes cases where humans tend to provide not only the description of video contents, but also high-level reasoning behind it. For instance, a boy is walking to the beach to surf versus a movie about a man who likes to surf. On the other hand, inconsistency in granularity consists of samples with human annotations at different graininess,e.g.people might coarsely describe the video as a cartoon character talking on the phone while a fine-grained description summarizes the same video with Krabs talks on the phone while SpongeBob freaks out. We argue that collective outliers are not impossible to learn. Instead, they can be exploited with more consistent descriptions or learned at a later stage of the training process when general principles from easier examples are already learned by models. Our observations and analysis align well with the philosophy of curriculum learning, encouraging our active learning design in the following paragraphs.Denoting the labeled set as L = {V m , C m } M m=1 , consisting of M video sequences and their human annotated captions C m , our video captioning model f is initially trained with L. We further denote another set of N unlabelled videos as U = {V n } N n=1 . Mathematically, U ∩ L = ∅. Our goal of active learning is to select a subsect S t ∈ U at each selecting step t ∈ {1, . . . , T } such that the overall performance of f can be maximally boosted once annotations on S t are obtained from humans.As stated above, collective outliers can be regarded as a group of hard-to-learn samples in an active learning framework where inconsistencies in human annotations are one of the main causes. Due to the lack of access to these annotations when selecting from U, we propose to mimic them to Together with uncertainty and diversity, our active learning scheme ranks unlabelled videos and parses them to humans. Our caption-wise protocol further provides an intellectual yet effective way to allocate human efforts, leading to 103% full performances at 25% human annotations.measure those inconsistencies on the mimicked annotations. In particular, we exploit LVLMs since they provide powerful tools to generate human-like captions at the same granularity. On the one hand, one can approximate the abstraction inconsistency among predictions from LVLMs. On the other hand, by comparing the predictions from f and LVLMs, granularity inconsistency can be further approached. Rather than leveraging foundation models for videos, we utilize the predicted per-frame captions from image-based LVLMs for the following three reasons. Firstly, the image-based LVLMs are more mature and well-exploited compared to the former. Moreover, inconsistency in abstraction occurs more frequently when video samples are complex, e.g., contents vary a lot temporally. Such variety can be better captured by measuring the consistency between frame-wise predictions. Lastly, our design provides a lower bound when measuring the inconsistency in granularity as frame-wise predictions from LVLMs and video captions predicted by f look at videos from different perspectives to the maximum extent, leading to more reliable approximations.To this end, we uniformly sample I = 32 frames from V n and refer to the i-th frame as V i n . Then we apply BLIP2 Li et al. (2023a) on all sampled frames, leading to a set of {b i n } I i=1 . b i n is the predicted caption on V i n . Meanwhile, we can obtain the video captions on V n by f (V n ) and further denote the one with the highest confidence as b * n . Inspired by SPICE Anderson et al. (2016) , we convert captions to scene graphs and measure the consistency over graphs. Specifically, we denote G i n = {O i n , A i n , R i n }= I i=1 O i n , A n = I i=1 A i n and R n = I i=1 R i n .In the next few paragraphs, we will introduce our active learning scheme that aims to capture the learnability, diversity and uncertainty in video captioning.Our scheme is composed of four terms. The first two of them focus on learnability, namely abstraction and granularity inconsistencies. The first term L 1 n focuses on the scene graphs generated by per-frame BLIP2, or G n , and measures their internal prediction consistency. Our L 1 n is defined as:EQUATIONwhere H k (G n ) counts the number of times that an object, an attribute, or a relationship k appears in G n . |O n | and |A n | capture the number of unique objects and attributes in G n , respectively. Intuitively, Eq. 7 prefers {b i n } i when they agree with each other. In other words, the higher the L 1 n is, the more consistent b i n should be w.r.t. predictions from other frames. We purposely neglect relations R i n in experiments as the relationship is less reliable in current LVLMs (e.g. BLIP2).Our second term L2 n focuses on granularity inconsistency where human annotations are at different graininess. To this end, we approximate the potential granularity inconsistency between G n and G * n as predictions from image-based LVLMs and that from video captioning model f tend to capture different aspects of V n to the maximum extent. In particular, granularity inconsistency can be measured by SPICE between two types of predictions, where a lower SPICE value indicates higher inconsistency. Instead of relying solely on SPICE, which tends to select well-learned samples and undermines the active learning scheme, we focus on the time-variant changes in SPICE for each V n to simulate the expected changes in granularity consistency. Denoting the absolute distance between the SPICE on an unlabelled sample V n at time step t -2 and t -1 as s n , we haveL 2 n = g( s n ) log g( s n ),where g is a min-max normalization function over all N samples. Our L 2 n prefers samples with moderate changes, based on our observation that large and small changes are associated with less informative and hard-to-learn examples, respectively.The third term L 3 n is designed for diversity. Specifically, we prefer unlabelled samples in which contents are beyond the current model f yet of great importance. In practice, we apply the concept of the Term Frequency Inverse Document Frequency (TF-IDF) Robertson (2004) to measure the importance of video content, in which we further incorporate our observation that longer descriptions tend to co-occur more with diverse videos by re-weighting the TF with long captions. Let's denote G * as the set that includes predictions of the highest confidence on L ∪ U, orG * = {G * m , G * n } N,M n=1,m=1 . Similarly, the BLIP2 predictions on full dataset is denoted as G = {G j } M +N j=1 = {G i j } I,M +N i=1,j=1 . Mathematically, L 3 n = F (O n ) + F (A n ) where F (•) is defined as: F (x) = k∈x I[k / ∈ G * ] H k (G n ) × log N + M N +M j I[H k (G j ) > 0](2)where I[•] is a binary indicator function and equals to 1 iff • is valid. Mathematically, L 3 n focuses only on contents beyond the current model. Meanwhile, it values distinctive samples with important contents. Overall, a higher L 3 n indicates greater diversity in V n . Uncertainty is captured by our last term L 4 n . Intuitively, inaccurate predictions provide valuable information in active learning. Therefore, we introduceL 4 n = |O * n ∩ O n | + |A * n ∩ A n | + |R * n ∩ R n |. Specifically, L 4n counts shared objects, attributes, and relationships between G * n and G n , where G n serves as human annotations to measure how well the prediction of highest confidence G * n is. A higher L 4 n reflects more certainty in V n . Finally, the overall active learning score on sample V n ∈ U is defined asL n = -λ 1 L 1 n + λ 2 L 2 n - λ 3 L 3 n + L 4 n ,where λ 1 , λ 2 , λ 3 are hyper-parameters. At the t-th step of the active learning algorithm, the top unlabelled samples S t from U are selected w.r.t. L n , where the lower L n signifies greater informativeness. These samples are then fed to human annotators to acquire annotations C St . Then we update our labelled and unlabelled set with L ← L ∪ {S t , C St } and U ← U \ S t . Later on, our video captioning model f is re-trained with the updated L. Our overall active learning algorithm iterates until either the maximum time stamp T or human annotation effort is reached.Conventional captioning-related active learning algorithms are video-based where all the human annotations from one video sequence are acquired if this video is selected at step t. In practice, there are multiple annotations associated with one video sequence, e.g., we have 20 captions for one video sequence in MSR-VTT dataset Xu et al. (2016) . We argue that the current video-based selection protocol is prone to collective outliers, leading to inferior active learning performance. Instead, we propose a caption-wise selection scheme such that not all annotations of one video sequence are acquired if this video has been selected. In practice, we acquire at most 2 2 captions for each selected video at the t-th step. Such a design reflects two of our observations. Firstly, the fewer annotations are acquired from humans, the less likely they are inconsistent with each other, leading to fewer collective outliers. Moreover, including more videos with fewer annotations rather than fewer videos with more annotations boosts the diversity of S t . The superiority of our protocol is shown in Sec. 4.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes] Justification: The code will be published after this paper is accepted. Guidelines:• The answer NA means that paper does not include experiments requiring code. Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 10. Broader Impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [NA] Justification: no societal impact of the work performed. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
experiment: 
 We validate our ideas with various backbones on two publicly available datasets MSVD Chen and Dolan (2011a) and MSR- VTT Xu et al. (2016) , and demonstrate SOTA performances on both.Datasets We conduct our experiments on two datasets, MSVD Chen and Dolan (2011a) and MSR- VTT Xu et al. (2016) . Specifically, MSVD consists of 1970 open-domain videos collected from a commercial video search engine, each of which is associated with about 41 human-annotated captions. Similarly, there are 10K open-domain videos in MSR-VTT and each of them has 20 human-labelled annotations. For each dataset, we follow their standard splits and report our active learning performance on their test sets. To mimic the learning process, we initialize L with 5% of data randomly selected from the training set, including both videos and their annotations. Consequently, U is composed of videos from the remaining 95% training set. At each selecting step, the annotation budget ∥C St ∥ is set to 5% captions of the full training set. More details can be found in the appendix.Baselines Active learning for video captioning is under-explored in literature. We follow the existing work Chan et al. (2020) and compare our algorithms with the baselines described below: • Random Sampling serves as a very competitive baseline in open-ended tasks. Basically, it performs random sampling in U to obtain S t . • Maximum Entropy Chan et al. ( 2020) is a conventional uncertainty-based active learning algorithm where samples with the highest entropy will be selected. In video captioning tasks, the entropy of sample V n is approximated by the averaged entropy over multiple predictions in f (V n ). For each prediction, its entropy is computed over word output distributions at each new word. • Minimum Likelihood Chan et al. ( 2020) is another uncertainty-based active learning algorithm.Typically, samples with the lowest log-likelihood will be selected. In practice, the averaged log-likelihood over multiple predictions in f (V n ) is used to rank each unlabelled video V n . • Core-Set Selection Sener and Savarese ( 2018) is a classical diversity-based active learning algorithm. Specifically, it works on the representation space and aims to select samples that are spread out in the feature space. In practice, features from Liu et al. (2022) are used to select samples that minimize the distance between the unlabelled pool to its closest labeled sample. • Clustered-Divergence Chan et al. (2020) ensembles multiple models and computes the KLdivergence between the conditional distributions of the ensemble members to measure sample uncertainty. Diversity is also implicitly considered as it already ensembles a clustering-based active learning regularization method. Implementation Details In our experiment, we employ SwinBERT Lin et al. (2022) and CoCap Shen et al. (2023) as our f as they provide good trade-offs between accuracy and efficiency. In contrast, other SOTA video captioning methods or video-based foundation models, such as COSA Chen et al. (2023) and mPLUG-2 Xu et al. (2023) , require extensive pre-training or finetuning when adapting to downstream tasks to achieve SOTA video captioning performances. To further speed up the training process, each video is uniformly sampled with 32 frames and these frames are parsed to f . The batch size is set to 4. We refer to the performance of f that trained with the entire training set on the test set as full performance. And we are able to achieve comparable performance compared to the official release. As for BLIP2 Li et al. (2023a) T is set to 4 . For other hyper-parameters, we keep the same configuration as in Das et al. (2022) . More details can be found in the appendix.Evaluation Metrics We provide detailed comparisons using a diverse set of performance metrics, including BLEU4 Papineni et al. (2002) , METEOR Banerjee and Lavie (2005) , ROUGE-L Lin and Och (2004), CIDEr Vedantam et al. (2015) , and SPICE Anderson et al. (2016) .We report the active learning performances of all methods with either SwinBERT or CoCap on MSVD in Fig. 3 . To ensure more reliable results, all methods were conducted three times. This figure reports both the average performance and the variance. There are several interesting observations. First and foremost, it is noteworthy that our method consistently outperforms all other baselines across all five evaluation metrics with two backbones, highlighting our superiority. Predictably, random sampling usually ranks as the second-best option, aligning with findings from other open-ended tasks Karamcheti et al. (2021) . More importantly, we observe that our method surpasses the full performance of f trained with 100% annotations with SwinBERT out of 4 evaluation metrics, as indicated by the dotted horizontal line, with less than 25% of annotations. For example, we achieve full performance with about 10% and 15% of annotations when measured by BLEU4 and ROUGE-L, respectively. Regarding CIDEr, our method attains 103.65% of the full performance with 25% of annotations. Even with CoCap, we can almost always achieve more than 95% of full performance with 25% annotations. This observation supports our concerns about the negative impacts of collective outliers during the training process. Clearly, our method effectively reduces the impact of collective outliers, leading to significant performance improvements over existing methods and even surpassing full performance. Results on MSR-VTT are reported in the appendix. In summary, our method significantly outperforms the SOTA methods and achieves an averaged 103% full performance with 25% annotations on all metrics.Cross-dataset Performance To simulate the scenario where people tend to exploit a large unannotated dataset to benefit a small annotated dataset, we use the small annotated dataset MSVD (1,200 videos with 40 captions per video) and treat MSR-VTT as the large unannotated dataset, which includes 6,513 videos paired with 20 captions each. The results using the official SwinBERT implementation are provided in Tab. 1. To demonstrate the effectiveness of individual components of our design, we conduct a thorough ablation study by incrementally integrating various components into our active learning scheme. The steps are as follows: 1) L 4 n (uncertainty); 2) +L 3 n (diversity); 3) +L 2 n (learnability); 4) +L 1 n (learnability); 5) + Caption-wise Protocol (CP); Specifically, we report the Area-Under-Curve(AUC) score over CIDEr and SPICE curve in Tab. 2. Evidently, incorporating any of these components will enhance overall active learning performance, demonstrating their effectiveness. For example, our designed terms outperform Random Sampling significantly, indicating the effectiveness of our active learning scheme without CP. Notably, we observe a significant performance boost after integrating the CP, which is reasonable as it directly reduces the impact of collective outliers and improves overall diversity. To further demonstrate that learnability, uncertainty, and diversity each reflect distinct aspects of active learning, we report the percentage of overlapped selections in S 1 under active learning schemes focused on individual aspects, as shown in Fig. 4 . As expected, our design targeting learnability, uncertainty, and diversity addresses various unlabelled samples, leading to limited overlap in their output S 1 . Together with Tab. 2, we conclude that all aspects are complimentary and mutually informative.Are we celebrating from more annotations per video? As summarized in Sec. 3.1, ground truth inconsistency is one of the main causes of collective outliers. We further showcase in Fig. 3 that fewer human annotations bring in better performance. Then a natural question to ask is whether we are celebrating from more annotations per video.To answer this question, we conduct a caption-wise random sampling experiment with SwinBERT on MSR-VTT. Specifically, SwinBERT is trained with all training videos on MSR-VTT where each video is associated with k ∈ {1, 3, 5, 7, 9} captions that are randomly sampled from a full set of 20 human annotations. We visualize the CIDEr and SPICE score on the test set of MSR-VTT in Fig. 5 .As shown in the figure, full performance is achieved with approximately 35% of the annotations (equivalent to k=7). Overall performance improves with up to nine annotations. We argue that the performance degradation with the full training set is not due to overfitting, as techniques such as regularization, data augmentation, and early stopping have been applied to mitigate it. Instead, we hypothesize that it is related to collective outliers, where inconsistency increases with the number of annotations. Additionally, we observe that captionwise random sampling performs worse than ours with the same amount of human annotations, highlighting the effectiveness of the L n design.Can we truly reduce the impact of collective outliers? Though we leverage the knowledge from LVLMs to approximate human annotations and thus identify collective outliers, it remains unknown how well such approximation or identification is. To address this, we first divide unlabelled samples into different groups w.r.t. their learnabilities, and then we obtain the distribution of S t according to these groups. Ideally, our method should select fewer samples that belong to collective outliers. Specifically, we divide unlabelled samples into four discretized groups according to their x, y coordinates in Dataset Maps in Fig. 1 . (b). Easy samples are those whose y > 0.07 and x < 0.03.Moderate ones fall into the region with y > 0.07 and x ≥ 0.03. For these samples whose y ≤ 0.07 and x > 0.17, we call them hard samples. The remaining samples are then collective outliers. We report the data distribution of S 1 of all methods in Fig. 6 . Again, results are obtained on MSR-VTT with SwinBERT as f . First and foremost, we observe that our method is able to select the least collective outliers compared to other baselines, which fulfills our goal as expected. Another interesting observation is that our method prefers to select more easy samples at S 1 . It is an effective strategy to select easy and moderate samples for learning in the early stage of model training, which greatly improves the efficiency and effect of training. By avoiding collective outliers and more reliable samples, it's no wonder ours achieves SOTA performance. We refer the readers to the appendix for more analysis.Can we exploit the knowledge from LVLMs more? Another way to utilize the predictions from LVLMs is to directly apply them as pseudo ground truths to update f in a semi-supervised learning setup. Specifically, this simple approach achieves 0.31 and 0.07 for CIDEr and SPICE on MSR-VTT, which are worse than the starting points. Adding additional control, e.g.including only videos when predictions from LVLMs are highly consistent, achieves 0.5185 and 0.0712 for CIDEr and SPICE, which is worse than our algorithm. We refer the readers to the appendix for more details.Are inconsistencies in ground truths our illusions? To validate our hypothesis that inconsistency in human annotations genuinely exists and is not merely due to subjective judgments, we utilize ChatGPT Radford et al. (2019) as an objective tool. We observe that ChatGPT believes that 49% of all captions are inconsistent on average. Such inconsistency rate increases when moving from easy to collective outlier sub-regions. More details can be found in the appendix.Does the reduction of human annotation costs justify the extensive computational cost? Our additional computational costs arise from the active learning algorithm, primarily due to applying BLIP2 to the unlabelled dataset and generating scene graphs from its predictions. This process occurs only once on the unlabelled set. On our hardware, consisting of 4 RTX 4090 graphics cards with a power capacity of 2000 kWh, it takes no more than 9 minutes to run BLIP2 and 5 minutes for scene graph generation on the MSVD training dataset. In contrast, annotating the full dataset in 2010 required hundreds of annotators, around 2 months, and less than 5000 USD in total Chen and Dolan (2011b) . Therefore, we argue that active learning is more efficient in terms of both time and cost.Limitations There are several limitations that we believe are worth further exploration. Firstly, our paper only briefly touches on the relationship between curriculum learning and learnability.Beyond provoking the design of the learnability term, we believe curriculum learning can enhance the interpretability of learnability terms and even active learning. Secondly, we found that current evaluation metrics, such as CIDEr, do not always align with human evaluations. More human analysis is needed for video captioning tasks. Thirdly, we made some preliminary attempts to combine knowledge from LVLMs in a semi-supervised learning manner. Although we did not see a significant improvement, we believe further efforts are warranted. Lastly, our experiments with ChatGPT-4 can be improved with more refined designs. We will include these limitations in our final version. 
conclusion: 
 In this work, we propose a novel active learning algorithm for video captioning tasks, which effectively leverages learnability, diversity, and uncertainty. To the best of our knowledge, our algorithm is the very first one that targets collective outliers in video captioning and further proposes to reduce their impacts by measuring sample learnability, as well as introducing a caption-wise protocol. Results on two datasets demonstrate the superiority of our algorithm over SOTA methods, e.g.we can achieve 103% of full performance with 25% of human annotations on MSR-VTT. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: the paper poses no such risks Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] the paper does not use existing assets.Justification: the paper state which version of the asset is used. The authors cite the original paper that produced the code package or dataset.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: In our appendix, we first share our quantitative results on the MSR- VTT Xu et al. (2016) dataset in Sec. A.1, followed by more details about our implementation in Sec. A.2. We further include our comprehensive analysis of the property of Dataset Maps, knowledge from LVLMs, and results with ChatGPT in Sec. A.3, Sec. A.4, and Sec. A.5, respectively. Our code and model will be made available.We report the overall performances on MSVD in Fig 7 where methods are visualized with different colors. As can be found in this figure, results on MSR-VTT share similar trends with those on MSVD. First and foremost, our proposed method consistently surpasses all baselines under all evaluation metrics. Among them, random sampling remains the safest choice as it almost always is the second-best. Both observations verify our motivation for tackling collective outliers in open-ended tasks. Remarkably, our proposed active learning method quickly achieves full performance using only a few annotations. For example, our method achieves more than 90% of the full performance under four evaluation metrics including BLEU4, METEOR, ROUGE, and CIDEr, with only 10% annotations. Moreover, it achieves 107% of the full performance under CIDEr, when only 25% of the full annotations are exploited. We would like to note that our hyper-parameters in L n are learned on MSR-VTT only and directly applied to MSVD, which further showcases the robustness and generalizability of our proposed method.As described in our main paper, we include two uncertainty-based active learning baselines as suggested in Chan et al. (2020) , or Maximum Entropy and Minimum Likelihood, for video captioning tasks. More details of them are provided in the following. word output distributions at each new word. Mathematically, the acquisition function of Maximum Entropy Chan et al. (2020) is defined as:EQUATIONAt each time step t, Maximum Entropy selects the unlabelled videos with high L n , reflecting the intuition that samples with high uncertainty are more likely to be informative.In contrast, Minimum Likelihood directly focuses on the averaged probabilities rather than entropy. Specifically, we follow the literature Chan et al. (2020) and define the acquisition function of Minimum Likelihood as:EQUATIONSimilarly, Minimum Likelihood favors unlabelled samples with low L n , which is well-aligned with the assumption that uncertain samples are more valuable.Post-processing Steps Scene Graph Parser Scene graphs are of great importance to our method as our learnability, uncertainty, and diversity are all measured upon them. Therefore, we describe the scene graph parser in length here. Specifically, We reused the pipeline of SPICE Anderson et al. (2016) to generate scene graphs from caption predictions, in which a Stanford Scene Graph Parser Schuster et al. ( 2015) is adopted. In particular, given a caption b i n or b * n , a Probabilistic Context-Free Grammar (PCFG) dependency parser Klein and Manning (2003) aims to firstly translate it into dependency parse trees. Then a semantic graph can be obtained by simplifying quantificational modifiers and resolving pronouns. After parsing the semantic graph through some simple linguistic rules, a scene graph G i n or G * n , which consists of objects, their attributes, and relationships, can be obtained. We provide an example of one caption and its scene graph in Fig. 8 .As formulated in the main paper, our acquisition function for the unlabelled sample V n is:EQUATIONwhere λ 1 , λ 2 , λ 3 are hyper-parameters. Let's further denote the annotation budget, or the amount of annotations to be obtained, at the t-th step of the active learning algorithm as A t . In our case,A t = ∥C St ∥.Video-wise Selection Under conventional video-wise selection protocol, once V n ∈ U is selected by active learning algorithms, all annotations in V n will be acquired. Assuming that each V n is associated with D captions, the video-wise selection protocol then selects the top ∥S t ∥ = At D videos from U and get all their captions. In practice, D equals to 20 and 40 in MSR- VTT Xu et al. (2016) and MSVD Chen and Dolan (2011a) . As a reference, we denote the annotations for the m-th labelled video V m as C m . Denoting the d-th caption as C d m , our C m can be represented asC m = {C d m } D d=1 .Caption-wise Selection Rather than assuming that all D captions will be acquired once V n is chosen by active learning algorithms, our caption-wise selection scheme allows more flexibility w.r.t. Eq. 5. Specifically, at each selection step, the video that has been least annotated with lower Eq. 5 will be chosen to get captions. Mathematically, we make the following revision:EQUATION)where C n denotes the number of annotations that already obtained on V n . And q is a dataset-specific hyper-parameter served as a re-ranking factor. Eq. 6 reflects our intuition that videos that have not been annotated are more informative due to both diversity and inconsistent annotations from collective outliers. Consequently, we will rank all V n according to Ln as long as |C n | ̸ = D with our caption-wise selection scheme.After ranking all V n that has not been fully annotated, we then start to acquire human annotations w.r.t. A t . Instead of equally allocating human efforts to top-ranking videos, which neglects the learnability property, we further introduce an intellectual design where more budgets are provided to topper-ranking videos. Compared to their peers, these topper-ranking ones are less likely to be collective outliers. Therefore it is safer and more effective to spend more effort on them. Specifically, we propose a stage-wise acquisition scheme by dividing the ranked videos into R consecutive and exclusive regions according to their L n . If the ranked V n belongs to the r-th region, where r ∈ {1, . . . , R}, then we allocate A t,r annotations to it. Let's denote annotations V n get at time t as C t n . In this case, A t,r = |C t n |. As a reference, we haveA t = n |C t n | and C n = t C t n .Our full caption-wise algorithm is summarized in Alg. 1.Algorithm 1: Our Caption-wise AlgorithmInput: The labeled set L = {V m , C m } M m=1 , the unlabelled set U = {V n } N n=1, a video captioning model f , a foundational model f ′ , the number of sampled frames I, the total active learning step T , number of stages R and stage-wise budget A t,r Output: Updated video captioning model f 1 Initialize f by training it on L; 2 Generate frame-wise captions {b i j } I,M +N i=1,j=1 with the foundational model f ′ ; 3 Parse {b i j } i,j into scene graphs G = {G i j } i,j ; 4 for t = 1 to T do 5 Generate predicted captions b * j with f for each video on U ∪ L; Hyper-parameter Setting We set the number of sampled frames I, the total active learning step T , and the per-step annotation budget A t to 32, 4, and 5% of full annotations in each dataset. The hyper-parameters λ 1 , λ 2 , λ 3 in Eq. 5 are 3, 1, and 2, respectively. And they are chosen based on experiments on the validation set via grid search. Our re-ranking factor q in Eq. 6 is set to 10 on MSR-VTT. Meanwhile, R is set to 3, dividing the ranked videos into 3 regions according to Ln . Specifically, both the first and last regions consist of 2000 videos. A t,r equals to 2, 1, and 0 with r = {1, 2, 3}. On the MSVD dataset, q and R are set to 20 and 5. These 5 regions in MSVD consist of 250, 250, 250, 188, and 262 unlabelled videos, respectively. With increasing r, we have A t,r set to 4, 3, 2, 1, 0.We would like to note that hyper-parameters related to the Caption-wise Selection Protocol are not optimal due to the lack of tuning. We might expect performance boosts with a more careful search.• people look from an overlook toward a hillside village and a lake below on a cloudy day. • an overlook that can see for miles lots of forest area to the left and a small town to the right.• a drone is flying over a green forest.• a man is flying over a lake and a city.• in a kitchen a man is frying meat in a pan turning it on the other side. • big meat pieces after cooking are fried in the pan care to take fry both sides.• a person is preparing a chicken in a pan.• an older man is cooking in the kitchen.Human-annotated GTs BLIP2 Predictions Unlabelled Video To the best of our knowledge, we are the very first to address the collective outliers in video captioning tasks. As described in our paper, we adapt Dataset Maps Swayamdipta et al. (2020) to perform analysis on collective outliers. We provide Dataset Maps on the training set of MSR-VTT in Fig. 9 .In particular, y and x axis are the averaged SPICE score over all training epochs and their variance. Intuitively, the average SPICE score reflects sample difficulty, while the variance represents its ambiguity. Combining both would provide us with a way to capture data learnability. Dataset Maps, to this end, provide an interpretable way to identify collective outliers w.r.t. data learnability.To verify our assumption that Dataset Maps helps in terms of figuring out collective outliers, we propose to divide samples according to their learnability. Specifically, samples are firstly divided into two halves based on the average SPICE score. We further conduct another baseline where f is trained with 25% randomly sampled data from the full training set. Compared to results from data with various learnability, we observe that this new baseline gives better performance. We argue that such inferiority comes from a lack of data diversity and uncertainty. As a reference, we also report the performance of our active learning method with 25% training data, which clearly gives the best performance with the same amount of data.Given the observation that LVLM predictions provide satisfactory approximations to humanannotated ground truths, one might wonder whether they can be directly used to either replace the human annotations or enrich them.Motivated by this idea, we conduct a simple semi-supervised learning experiment where the initial setup is the same w.r.t. our active learning scheme, e.g., 5% and 95% split for labelled and unlabelled sets. For each video in the unlabelled set, we obtain its pseudo ground truth from BLIP2 by randomly selecting 16 captions from 32 frame-wise predictions. We then update our f with these pseudo ground truths and report its performance on the MSR-VTT test set in Fig. 10 . Apparently, there exist clear differences in language style between captions generated by BLIP2 and human-annotated ground truths. Specifically, this simple semi-supervised learning method achieves 0.31 and 0.07 for CIDEr and SPICE, which are worse than the starting points. Such performance drop might be caused by domain gaps between video and image captioning tasks, as well as the lack of ability to capture temporal information.Can the gap between human-annotated video captions and predictions from image-based foundational models become smaller if the V n is so simple that video caption degenerates to image caption? Since there is no way to recover the temporal information from frame-wise image caption, we propose to address this factor from another perspective. Specifically, we believe that the impact of temporal information can be minimized if videos are highly coherent among frames. Such coherency can be further approximated by frame-wise consistency. To this end, we propose to include videos whose frame-wise BLIP2 captions are of high abstraction consistency. Mathematically, let's denote P (b i n ) as follows:EQUATIONThen the abstraction consistency of each BLIP2 caption in unlabelled videos is defined as follows:EQUATION)As long as its score P (b i n ) is not lower than a threshold th, a BLIP2 caption will be counted as a pseudo ground truth. Later on, these pseudo ground truths will be used to re-train f , which is initialized with 5% of training data only. We report the performance of f in Tab. 4 with th ∈ {4, 8, 12, 16, 20, 24, 28}. Noticeably, CIDEr score increases with the increasing th. This observation validates our hypothesis that videos of highly consistent content generally have a smaller domain gap in terms of video and image captioning, thanks to the neglectable impact of temporal information. In contrast, including more captions generated by BLIP2 brings higher SPICE. This is mainly because of the semanticcentric property of SPICE. In short, captions that are semantically similar but grammatically different will be regarded as correct ones in SPICE.Since semi-supervised learning is beyond the scope of our paper, we do not include more experiments in this section. Though not observing performance improvements by introducing BLIP2 predictions as pseudo ground truths, we still believe that domain adaption methods with much more delicate designs would help. We look forward to seeing more work in this direction.To validate our hypothesis that inconsistency in human annotations genuinely exists and is not merely due to subjective judgments, we utilize ChatGPT as an objective tool.Specifically, we randomly sample 500 videos from each sub-region from MSR-VTT, and then randomly select 2 ground truth captions for each sample. We then ask ChatGPT whether it believes these 2 captions describe the same content. ChatGPT will respond with one of three answers: yes, no, or undecided. There are two main observations. First of all, we notice that ChatGPT can provide reasonable answers to our provided examples. Here we provide one pair of examples for each answer from ChatGPT:• Yes: "two teams play soccer on a field" & "a man gives commentary for a soccer game" • No: "there is a man talking in front of his computer" & "there are some tips to use your computer" • Undecided: "a video showcasing a modded jeep" & "a person gets his lifted jeep stuck in the dirt"Noticeably, based on the feedback of ChatGPT, there are 49.5% and 22.5% of inconsistent data and hard-to-decide among all 2000 samples respectively, indicating the severity of inconsistency in human-annotated ground truths. Moreover, when comparing the percentage of data among subregions, we observe a clear trend of consistency increases when moving from collective outliers to easy. For instance, 22.4% and 33% of captions from easy and collective outliers are believed to be consistent. This observation aligns well with our arguments and supports our hypothesis as well as motivation.NeurIPS Paper ChecklistThis number is chosen by experiment.Checkout the pre-trained model at https://huggingface.co/Salesforce/blip2-flan-t5-xl 


Decision: The paper proposes a new active learning algorithm for video captioning, which considers learnability, diversity and uncertainty to select unlabelled data. Experiments on two video captioning datasets demonstrate that the proposed method outperforms prior active learning methods in this domain.

Reviewer ZEo9 initially raised concerns about the overclaim of performance, low baseline performance and the annotation cost. Reviewer fZYA highlighted the lack of cross-data experiments and limitation analysis. Reviewer JoyP questions the use of LVLM, computational costs and the overall novelty of the method. The rebuttal addressed most of these concerns, although experiments on larger sota models could not be completed within the limited rebuttal period - a point acknowledged by Reviewer ZEo9 as a reasonable constraint.

After rebuttal, all reviewers maintained positive ratings with 2 BAs and 1 Accept. The AC agrees with the reviewers’ recommendations and suggests accepting the paper. The authors should include the necessary revisions in the final version as promised in the rebuttal.

Reviews: review: paper_summary: paper_summary: This paper works on active learning for video captioning, i.e., filtering the training data for a video captioning dataset. The authors observed significant inconsistency in the captioning annotation, due to different captioning abstraction or granularity. These inconsistency makes the model training hard. The authors then propose an active learning framework that find captions that the model can learn, combining with diversity and uncertainty metrics. The overall model trained with 25% of the annotated captions outperforms models trained with all captions on MSVD and MSRVTT.
strength_weakness: strength_weakness: - The authors only report relative performance with respect to the fully supervised baseline in the main paper, which "hides" the absolute metrics of the proposed model. Only in the supplementary Table 2, the authors report the absolute metrics on MSRVTT, which seems to be significantly below SOTA on paper with code. E.g., CIDEr of the best performing models are > 70, and the proposed model is at ~55. While I am not saying the paper needs to get SOTA performance, but this gap towards SOTA is concerning. I.e., it might be that more powerful model has less learnability issues, and the proposed model may gain less improvements. This also makes the claim in the abstract "our algorithm outperforms SOTA methods by a large margin" an overclaim.

- From Section 3.3, the authors select X% of the total captioning, rather than X% of the overall videos. Arguably, annotating two captioning for the same video can be more efficient than annotating two videos, due to shared workload in watching the video. This makes the claim of " 25% of human annotations" questionable.
questions: questions: Overall this paper studied a very interesting observation in video captioning (inconsistency in human annotation), and propose a valid framework to utilize this observation, with reasonable gains on selected baselines. My main concern is on the low baseline performance, resulting in overclaims on performance. My current rating is a boderline accept, conditioning on the authors response to the baseline issue and willing to rephrase the overclaims.
limitations: limitations: Not discussed.
paper_summary: paper_summary: The paper presents a groundbreaking exploration of collective outliers in video captioning tasks, introducing innovative active learning algorithms and an effective caption-wise learning protocol that integrates human knowledge to enhance model performance. Despite its strengths in pioneering research and sophisticated methodologies, the paper falls short in conducting cross-dataset experiments beyond MSVD and MSR-VTT, limiting its real-world applicability. Additionally, a lack of thorough limitation analysis and minor writing issues, such as visibility concerns in figures, suggest areas for improvement in future research efforts.
strength_weakness: strength_weakness: - Lack of cross-dataset experiments. The experiments are only conducted inter MSVD and MSR-VTT. This does not solve the challenges of video captioning in real scenarios. Given a small annotated dataset (like MSR-VTT) and a large unannotated dataset from the web (like WebVid or VATEX), what samples should we annotate from a large web dataset to improve performance on the small annotated dataset?

- Lack of limitation analysis.

- Some writing issues. (1) The fonts in Figure 5 can be hardly seen. Better keep font size in all figures the same.
questions: questions: See Weaknesses.
limitations: limitations: See Weaknesses.
paper_summary: paper_summary: This work proposes an active learning algorithm via data learnability and collective outliers. The algorithm takes three complementary aspects, learnability, diversity, and uncertainty, into account. The proposed algorithm leverages off-the-shelf models (e.g., LVLM) for approximations to ground truths.
strength_weakness: strength_weakness: The usage of LVLM is debatable. If the LVLM is sufficiently powerful to generate accurate captions on videos, training a captioning model via active learning seems unnecessary. If the LVLM cannot generate accurate captions, using the output from LVLM for different components of the acquisition function is unreliable.

The proposed framework requires extensive usage of LVLM and the scene graph, which incurs significant computational costs. On the other hand, many tools, such as Mechanical Turk, are available to collect human annotation cheaply.
I would like to know if the reduction of human annotation costs justifies the extensive computational cost.

The technical innovation for each component appears incremental, and the terms in the acquisition function are heuristic, without mathematical proof.
questions: questions: Please refer to the Weaknesses.
limitations: limitations: The authors adequately addressed the limitations in the checklist.

Comments: title: Thanks for your response.
Thank you for your rebuttal
comment: We are deeply indebted to you for the time and energy you devoted as a reviewer for NeurIPS 2024.  Your insightful feedback has been instrumental in enhancing our paper, and we are truly thankful for your acknowledgment of our efforts. If you have any further concerns, please don't hesitate to reach out.

Best regards,

The Authors
My concerns are well-addressed.
We sincerely thank you for your time and effort as a reviewer for NeurIPS 2024. We appreciate your engagement and the valuable feedback that has helped us improve our paper. We are grateful for your recognition and support of our work. If you have any additional concerns, please feel free to let us know.

Best Regards, 

The Authors
Thank you for your thoughtful feedback and for taking the time to review our rebuttal. Your valuable suggestions have greatly contributed to improving our manuscript. We appreciate your understanding of the limitations during the rebuttal period and welcome any further concerns or questions you may have.

Best Regards, 

The Authors
Thank you for your invaluable efforts and constructive feedback on our manuscript.

As the discussion period nears its conclusion, we are eagerly await your thoughts on our responses. We sincerely hope that our response meets your expectations. Should there be any remaining concerns or if further clarification is needed, we are fully prepared to address them as soon as possible.

Best regards,

The Authors
Thanks the authors for the rebuttal. Please make sure to add "SOTA **active learning**" in the main claim. While the authors explained why the baseline numbers are lower than the actual SOTA, it is still unclear to me why these particular baselines (SwinBERT and CoCap), rather than a more well-pretrained baseline model, are selected for experiments. I also understand it is not practical to re-do large scale experiments during the rebuttal period.

The authors response on annotation costs make sense.

Overall, my concerns are partially resolved. I'll keep my current borderline rating.
I thank the authors for preparing the rebuttal. After reading the rebuttal, I intend to keep my original rating, which is positive.