Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes an approach for automatically selecting a few frames from training videos to be annotated by humans and for using these annotations to train models for spatio-temporal video understanding tasks, such as spatio-temporal action detection and video object segmentation.

1. In contrast to prior work on active learning strategies for video understanding, which choose which videos to fully-annotate, the proposed approach chooses specific frames under a specific annotation budget.
2. The first contribution is an approach for active sparse labeling in videos, which utilizes frame-level uncertainty of the model to identify frames that need to be annotated. Motivated by the task of action detection, it also ensures that a) a set of temporally diverse frames are selected (instead of consecutive frames with redundant information), b) that background pixels don’t influence the selection (since the model might be fairly certain for them). 
3. The second contribution is a training regime from learning from sparsely-labeled frames, which involves interpolating annotations in the rest of frames, and a loss (max-gaussian weighted loss) that discounts the penalty from wrong predictions at frames that are distant from the ones that have human annotations.
4. The proposed method is evaluated on three datasets: UCF-101 and J-HMDB for spatio-temporal action detection, and YouTube-VOS for video object segmentation, where it is shown that it outperforms other baselines for frame selection given a fixed annotation budget.

strengths_and_weaknesses: Strengths
========

1. The paper addresses an important challenge in the action detection literature, namely how to train systems with fewer labeled data. This is especially important for tasks that require fine-grained annotations, such as bounding boxes or segmentation masks, which is the focus of this paper. Focusing on spatio-temporal video understanding tasks and on selecting frames for annotation instead of whole videos clearly distinguishes this work from prior approaches. The review of prior approaches also seems adequate.
2. The proposed approach is simple, but well-motivated (both intuitively and based on ablations). It builds upon MC-dropout for uncertainty-based active learning, but also takes into account the nature of videos with the proposed Adaptive Proximity-aware Uncertainty, which outperforms other uncertainty-based selection methods (Fig. 4).  The proposed loss also outperforms training simply with interpolated annotations or only on annotated frames (Fig. 5).
3. The proposed approach outperforms other frame selection baselines (such as choosing equidistant frames or using approaches that were proposed for images).
4. The paper contains multiple qualitative examples, which clearly demonstrate the issues of related baselines. 
5. The proposed approach does not require an actor/object detector.

Weaknesses
===========
1. Since this seems to be the first approach for active learning for these particular video understanding tasks (spatio-temporal action detection, video object detection), choosing the right baselines (in Table 1) is crucial. The evaluation could be strengthened by comparing with video selection methods, such as approaches from “What do I Annotate Next? An Empirical Study of Active Learning for Action Localization”[60].
2. Method is only evaluated for selecting frames up to 9-10% frame annotation percentage: It would make sense to investigate how many annotated frames are needed to reach the fully-supervised performance (if possible).
3. Comparison with state-of-the-art: Table 3 is comparing methods with very different levels of supervision and architectures. Although it is nice that the proposed method leads to the best performance, it should be clear in the table that the comparison is not apples to apples. It would be helpful to add more details about the type of annotations (points?, temporal bounds? bounding boxes?) and the performance of each method given full-supervision if available (this will help clarify whether the models trained were much weaker than the capsule-based network). For completeness, the state-of-the-art fully-supervised method for each dataset should also be included.
4. More details about the training loss (margin-loss, binary-cross entropy loss) and the metrics (do they refer to temporal IoUs, spatio-temporal IoUs etc) are needed.


questions: Questions for rebuttal
1. How were the videos selected in Figure 6 (c-d)? Comparing with active learning methods for video selection would strengthen the paper.  It would also be feasible to try hybrid approaches, e.g. choose videos samples with active learning and then use equidistant frames within them, to fit more diverse frames within a given annotation budget.
2. Additional graphs: It would be nice to extend some graphs to more than 9-10% of annotated frames. It would also be interesting to see the performance over the different steps of active learning, as more frames are added.

Suggestions
1. What is the impact of \sigma for the MGW-Loss?
2. How many active learning cycles are used?
3. What happens if you start from the same percentage of annotated frames in UCF-101/JHMDB?


limitations: Yes
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposes an efficient learning paradigm for the task of video action detection where all the frames in a video need not be labelled with spatio-temporal annotations. The task is important since the cost of annotations increases as the task granularity increases from action classification to action detection. The paper uses an active learning scheme to select most informative frames to annotate, and manages to achieve the performance comparable to fully-annotated approaches while using a fraction of the labelled frames

strengths_and_weaknesses: ## Strengths
- The task itself is important and the conclusions are worth looking into for any action detection approaches
- The approach is simple, intuitive and most likely general enough to be deployed in the setup of fine-grained video tasks

## Weaknesses
- Dataset and baseline in Table 1 and 2
    - The random baseline method achieves f-mAP@0.5 score of 69.3 with 10% data as compared to the proposed approach’s score of 71.7
    - Given the claim that the proposed approach reduces the need for annotation by 90%, the baseline itself seems to be not that far away, do the authors have an estimate of how much annotation does the baseline method need to achieve the performance of fully-supervised approaches?
    - If the answer is, say less than 50%, does that say anything about the dataset itself? Do the authors think that experimenting with more complex dataset might help bring out the efficacy of their approach?
- Non-activity suppression and relative area of spatio-temporal predictions (para on line 154)
    - The spatio-temporal annotations should have different relative area that they occupy in a frame, i.e, they lie on a spectrum same as MS-COCO object bounding boxes
    - Since the approach needs to contend with uncertainty being influenced by background, do the authors have some breakdown on performance w.r.t. relative area of spatio-temporal annotations? This is similar to AP_S, AP_M and AP_L evaluation procedure in MS-COCO object detection task
    - The reason for asking this is to check whether the algorithmic choice in the paper induces some preference over the relative area of spatio-temporal predictions
- Annotation cost of far-away frames v/s nearby frames in a video?
    - The analysis made in the paper assumes a uniform frame annotation cost regardless of where the frame is located w.r.t. annotated frames
    - My guess is that, in real life, annotating chunks or blocks of frames is easier than annotating frames one-by-one far away from each other in a video. This suggests that the cost of annotating frames will be lower near the annotated frames, or the selection process can actually select a block of frames at a given location without sacrificing too much of the cost
    - This argument partially contradicts the design choices made in the paper about not choosing proximal frames using Gaussian kernels
    - Do the authors have any comment on the above? Did the authors perform real-life analysis of annotation cost on the UCF and JHMDB datasets and validate their assumption that annotation cost of a frame is same regardless of where it occurs in a partially annotated video?

### Low priority
- Use of pre-trained weights on Charades dataset (line 215)
    - Do the authors have any intuition regarding the selection of Charades dataset? Is the approach expected to perform differently with different pre-training datasets such as Kinetics?
- Scalability of the approach to larger datasets
    - Did the authors experiment with larger datasets to check whether the approach generalizes with scale of the datasets?

### Nice to haves:
- It would be nice to have the word “utilize” substituted with “use” throughout the text. It will help reduce cognitive overload.
- The supplementary has a lot of sections, but are not cross-referenced in the main paper. It would be nice to have that so the reader can go into details on a particular section if they want to

questions: See "Weaknesses" above
limitations: Yes
review_title: null

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: This paper presents an iterative frame-selection approach to select a subset of the most typical/useful frames from all video frames for reducing the annotation cost for the task of video action detection. In particular, the paper introduces a frame-level scoring mechanism in terms of pixel uncertainty in a video aimed at selecting the most informative frames in the video. In experiments, the proposed approach was evaluated on action detection benchmark datasets UCF-101-24 and J-HMDB-21.
strengths_and_weaknesses: Strengths 

1. This paper works on the problem of active frame selection for the task of video action detection.

Weaknesses

1. The novelty seems incremental. The definition of frame uncertainty for selecting video frames as well as the max-gaussian weighted loss for training the action detection network is either taken from or extended from the existing approaches. The paper does not provide any new insights. 

2. The paper settings seem unfair. From the settings, seems like the proposed approach only works for easy datasets with a single object on a static background but not for complex video scenarios. Working on the problem of active frame selection in a simplified/reduced constraint setting seems to be unjustified. Also, the comparison with respect to the state-of-the-art approaches seems to be unfair. For example, this paper uses the I3D encoder head with pre-trained weights from the Charades while [23] uses the I3D network trained on the Kinetics dataset.

questions: See above section 'Weaknesses'
limitations: The paper seems incremental and the experiment setting seems unfair.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper describes a novel training paradigm for action detection in videos.

Technically, the approach consists in an sparse labelling model that implements a frame-level scoring module that tries to select the most informative/discriminative frames for action detection. With this sparse labels, the paper describes a training approach, with its associated loss function. 

An experimental evaluation is performed in two publicly available datasets, and the results reveal the benefits of the sparse active labeling technique.
strengths_and_weaknesses: # Strengths

- Writing a scientific article is not easy, but writing it well is really an art. This manuscript has been written in a careful and engaging way for the reader. Ideas are not masked in confusing paragraphs, but are clearly explained. It's a pleasure to be touched to review articles like this. I thank the authors for their efforts.

- Section 2 adequately discusses previous works, showing the novelty of the proposed approach. Overall, the application of active sparse labeling for training action detection models in videos has not been previously explored.

- The experimental evaluation follows clear experimental setups using publicly available datasets (UCF-101-24 and J-HMDB-21). The comparison with state-of-the-art models is fair and sound.

- Active Sparse Labeling idea is worth to be shared with the rest of the computer vision community. The way this paper treats the sparse labels could be of interest for continual learning approaches, where models face to new tasks for which no annotations (or a few) could be at hand. Moreover, the learning with the Max-gaussian weighted loss could be applied to other semi-supervised pipelines, where the loss can offer a mechanism for assigning to the pseudo-labels a sort of confidence measure.

- The experimental evaluation is well designed: specific experiments to validate the contributions claimed in the paper. It also reports comparisons with state-of-the-art semi-supervised methods, and a thorough ablation study. The paper defines the state-of-the-art for the semi-supervised setting on the problem, with just 10% of the frames being annotated with ground-truth.



# Weaknesses


- As it is pointed in the manuscript, as an AL model, scalability is a weakness of the model. We need multiple iterations to select the frames, and this is time consuming. It would be fundamental to know how much time needs the deep learning architecture between iterations.

- I found some technical limitations that are worth to be discussed:
a) It was not clear to me why does the model need an estimation of the uncertainty per pixel, that has to be obtained utilizing MC-Dropout. To run Eq. 1 should be highly time-consuming. Are there any other uncertainty measures that could be used? Have the authors used any alternative?
b) Adaptive Proximity-aware Uncertainty (APU) objective is to select frames with temporal diversity, am I wrong? Eq. 3 presents a combination between the distance and the uncertainty with just a sum. Are both variables scaled? With a lambda of 0.5 the model assumes that the two terms are in the same range of values. An ablation study on the influence of lambda could be interesting.
c) In the Non-activity suppression block, I wonder the influence of tau in the performance of the model.
d) It is unclear to me how the proposed model is integrated in VideCapsuleNet [9] approach. Section 3.e needs to be extended.

# Minor comments:
- Please, punctuate all the equations. They are part of the text.
questions: This is a paper that may be of interest to the computer vision community. The manuscript is well written, and the ideas they propose are novel (there is no similar work using AL for action detection in videos). The experimental results are convincing and allow us to judge the contributions claimed from the paper. In addition, the experimental evaluation developed is thorough and critical. Overall, except for some minor shortcomings that I would like the authors to address, I believe the article should be accepted.
limitations: Yes. The authors identified as the main limitation of the approach that as most AL is a time consuming model. Clarifications about the runtime are needed, as I pointed above.
review_title: null

Meta-review: metareview: Paper was reviewed by four reviewers and received: 2 x Weak Accepts, 1 x Reject and 1 x Accept. Reviewers argued that the task is interesting and approach is simple and well motivated. Some raised concerns included: (1) lack of additional comparisons and baselines, (2) lack of discussion regarding far-away frames v/s nearby frames for annotation, (3) lack of novelty and (4) fairness of evaluation. Three out of four reviewers were reasonably convinced by the rebuttal and argue for acceptance. [2s1J] remains concerned about (3) and (4). This was carefully considered by AC. Because no specific papers were provided by [2s1J] to support the claims of lacking novelty, and given the remaining positives reviews, AC is inclined to accept the paper.  
confidence: Certain
Decision: Accept