Venue: NeurIPS 2022 Track Datasets and Benchmarks

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: In this paper, the authors propose a point tracking dataset with 1189 real videos. Three baseline methods are designed to evaluate the performance.
strengths: 1. The research field of video TAP is under-studied. This paper introduces a companion benchmark it would benefit the community.
2. The data collection method provided in this paper shows a standard pipeline for the following research.
3. The proposed baseline method presents the usability of the benchmark.
weaknesses: 1. The method could not be used for transparent objects.
2. The synthetic data may not lead to bias in model evaluation.
correctness: The claims appear to be correct.
clarity: The paper is well written.
relation to prior work: Prior works are compared with the proposed dataset and the difference in scale is highlighted.
additional feedback: No additional feedback.
review_title: Review for TAP-Vid

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: summary and contributions: The paper proposed TAP-Vid dataset, which is a benchmark for tracking any point in a video. Given a video clip and a set of query points, the goal is to predict trajectories over the whole video whether it is occluded or not. The benchmark contains four parts of data which are basically sources of videos: Kinetics, DAVIS, Kubric and RGB-Stacking. Two of them are real videos and the others are synthetic. Synthetic videos provide a good training set, while real videos can be both useful in training and validation. For each video, 25 query points are sampled and the trajectories are manually annotated. Baselines have unsatisfactory performance on TAP-Vid dataset and suggests a lot of space to improve.
strengths: - The problem setting is interesting and very challenging for existing algorithms. It opens new research questions and I believe there will be a lot of follow-ups to improve the tracking performance on this benchmark.
- The annotations have high quality. I check the webpage and each subset has some sample videos (thank you!). The quality is beyond my expectation. They are very useful.
- The scale of the dataset is large. Given that each video clip is 10s, 532 real videos are a lot for annotation of the trajectory over all frames. Two synthetic subsets are also proposed to help the training.
- Experiments are extensive. Three types of baselines are proposed, as well as a novel approach TAP-Net.

weaknesses: I don't see any obvious weakness in this paper.
correctness: Yes.
clarity: Yes.
relation to prior work: Yes.
additional feedback: N/A
review_title: Review

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: TAP-Vid aims to expand keypoint tracking in video with a greater diversity of points and longer duration point tracks that account for occlusion. TAP-Vid introduces data, annotations, a benchmark, and a method for long-term tracking of points on solid objects in videos.

The contributed data is data recorded from a pre-existing robotic object-stacking simulator.
The contributed annotations are tracks of points on the surfaces of objects across four datasets: Kinetics, DAVIS, MOVi-E, and the RGB-Stacking robotic simulator.
The contributed benchmark consists of 3 evaluation metrics: occlusion accuracy (binary per point per frame occlusion), position accuracy (unscaled PCK), and a jaccard measure.
The contributed method produces features spatially for frames, selects the feature (or feature interpolations) that corresponds spatially and temporally to a given query point, then produces a matrix product between this feature and other features at the target time to produce a cost volume. This per-timestamp-pair per-qeury cost volume is then fed into a network that predicts a binary occlusion logit and a predicted (x,y) point that corresponds to the location of the query point in the target frame.

TAP-Vid is motivated by the data gap that exists between dense per-pixel tracking in short-term optical flow and sparse long-term keypoint tracking in video. The authors aim to fill in this gap with the above contributions.

TAP-Vid hypothesizes that if videos had more annotated point-tracks (including point-tracks that survive occlusion), then motion understanding in video would improve. TAP-Vid tests this claim by training a network, TAP-Net, that uses these new point-tracks. TAP-Net’s performance is generally improved when using these point-tracks vs. point-tracks from the existing JHMDB dataset. This analysis is slightly, but bearably, confounded by the differing definitions of keypoints that exist between human pose estimation and their surface point tracking.
strengths: 1. TAP-Vid's cross-dataset annotations allow for evaluation of generic motion tracking. Generic methods for motion tracking should work across datasets and domains. Since TAP annotations span multiple datasets, the benchmark is promising as a way to unify tasks and evaluations for general motion tracking in video. Existing dataset-specific evaluations may not be sufficiently expansive to evaluate generic motion tracking methods. This improves the significance of the contribution, as it paves the way for evaluating more generic methods.

2. TAP-Vid evaluating with points means it's a general benchmark for systems that might natively predict boxes/segments/etc. Point tracking naturally supports higher-level evaluations. Since TAP annotates point tracks, TAP evaluations can be run even on methods that only predict sprites/masks/canonical surface maps/other representations. This improves the relevance to the broader research community and accessibility by other researchers, as point tracking is a good interface for general tracking tasks regardless of method.

3. TAP-Vid builds on top of existing, valued, frequently-used datasets. Annotations of an existing dataset are useful in that they allow for more robust evaluations alongside tasks that are already valued. Since TAP provides annotations on Kinetics and DAVIS, it contributes to popular, in-use datasets. This increases the relevance to the broader research community, as the evaluation can be adopted to improve the evaluation richness of current methods.

4. TAP-Vid attempts to solve keypoints vs. joints. TAP-Vid rightly points out the conflicting definitions of keypoints depending on whether they refer to joints interior to a surface (as in human pose estimation) vs. whether they refer to surface points (in surface mapping and 3D tasks). A strength TAP-Vid is an attempt to move towards a clear, unified definition of point tracking while including the data and annotations that make such a new evaluation possible (allowing potentially for progress past JHMDB as shoehorned tracking testbed).
weaknesses: 1. The major weakness of TAP-Vid is that annotation quality means "tracking any point" (a good goal) turns into "watch all the failures in annotating hard to track points". The major weakness here is the quality of the annotations. In 5/6 of the DAVIS videos I observed, there were deviations in a track from what should be the ground truth. This is a fixable problem, but the annotation quality is cornerstone to the validity and usefulness of the benchmark.

I looked at a few visualized RGB-stacking videos and the first six visualized DAVIS videos by index in the visualize.py script. Here are my notes from them:

DAVIS 0 – teal point, hair on the back of the goat – jumps around the goat significantly although clearly the point to track is the hair it was indexed on.
DAVIS 1 – orange point, side of the ladies face in the car – jumps to the front of the glasses before jumping back to the side of the head.
DAVIS 2 – green point, front of bike fender – jumps to below the bike and then above it.
DAVIS 3 – orange point, background ladies jeans – migrates to the elbow of the guy wearing the black sweater.
DAVIS 4 – lack of diversity of points, mostly positioned on the two ladies in the far background (not a quality issue)
DAVIS 5 – green dot, rear left quarterpanel BMW – migrates a foot or so along the car
DAVIS 6 – yellow dot, forehead right man – looks like the dot moved with the man and then back with a delayed reaction, moving it down below the ridge of his eyebrows briefly.
RGB-Stacking – these points are probably perfect due to their synthetic nature. It is interesting to notice points flicker in and out of occlusion.

To resolve this major weakness, the authors can do an annotation quality analysis in comparison to alternative keypoint annotations in video, like JHMDB (although they annotate joints). If they find the annotation quality is superior to JHMDB then this dataset is a net positive to the community. If the quality is lower, then this work risks introducing noise to learning to track keypoints over time. I currently think that Table 3 in the supplementary material is not enough to resolve this issue for me, as it is unclear whether the accuracy differences are do to predicting joints or predicting keypoints. 

2. TAP-Vid creating "point-precise" annotations increases the stringency for "doing a good job". The “point-precise” nature of TAP and its annotations is a central motivator of the work, as expressed in Table 1. The difficulty here is that by choosing such a stringent task, it also significantly increases the quality expectations for succeeding at annotating this task. Given that a central argument is moving away from KEYpoint tracking to tracking any “trackable” points, it may be the case that annotators need more guidance on choosing trackable points, as they don’t seem to always succeed as is.

To resolve this weakness, resolve weakness 1 above. This weakness explains my perspective that there is self-imposed quality issues and to hint at where this issue might arise. It also offers an alternative resolution that is related to significant figures. If instead of "points" annotations are referred to as "blobs" and provided with spatial uncertainty estimates, it could reduce the stringency of discussing "points". 

3. TAP-Vid does not ablate the value of concerning components in their assisted annotation suite. I am concerned that the annotation suite with track assist, an important part of the paper, may actually contribute to the poor quality. Although the supplement includes an analysis of the “improved accuracy” of using the optical flow track assist, it doesn’t include an ablation of the annotation suite. Specifically, the annotation instructions steps 4 and 5 and 6 seem like they always end with an invocation to the “track assist”. I am concerned that from the pseudocode there is no escape path that lets you “skip” the “track assist algorithm”. If true this means that an annotator doesn’t have the “final word” and always has to move on to a new point after the “track assist algorithm” has "refined" a point. It may be that the iterative refinement pseudocode is not included here.

To resolve this weakness, I would like clarity on the iterative refinement and whether or not every point is truly forced to be refined by "track assist". This weakness is also related to weakness 1 and could be resolved by improving/analyzing the quality of the data. 

4. TAP-Vid's automatic tool is central to producing the annotations, yet due to how people respond to super accurate tools, the dataset isn’t really annotated in practice. Annotation speed is a potential cause for concern and source of quality problems. Annotation speed is important for the affordable creation of large-scale data. Assisting annotation for speed is valuable and done in this paper. From the paper, it seems that 2 annotator hours might go into tracking 30 points through every frame of a 25 FPS 10-second video clip. This seems to mean that 7500 (x,y,t) pairs will be annotated, at a rate of 62.5 annotations per minutes or roughly 1 annotation per second.
I am concerned that given the speed, the number of videos (over 1,100 videos from Kinetics) and the assist, it may be the case that an assisted tool that is correct 95% of the time is being rubber stamped by annotators. When automatic annotation tools become good enough, it can become the case that points are no longer explicitly checked and instead assumed to be correct.

To resolve this weakness, resolve weakness 1, etc. However, this weakness offers an alternative resolution -- avoiding or at least experimenting with annotation similarity with or without an annotation tool. If it turns out that a high percentage of no-tool-involved annotated point tracks are very similar to a sample of point tracks already annotated, it could similarly resolve accuracy concerns. 

5. TAP-Vid's annotation instructions bias annotators towards certain object parts. The annotator guidelines bias annotators towards the example part tags “human hand, human head, car wheel”. This may be the reason behind the biased choice of points from DAVIS 4 above. The authors do not address or analyze the above bias that may be introduced in point selection by annotators. It may be the case that annotators only select certain points on certain objects, but this potential semantic bias is not discussed as the motivation of the paper is to annotate points regardless of their semantic class. 

This is a minor potential/unexplored weakness in annotation quality related to semantic class and could be resolved by a semantic analysis of annotations.

6. TAP-Vid annotations may be systematically worse for deformable objects. Deformable objects are notoriously difficult to successfully track, yet they are an important part of any scene. Given the failure in DAVIS 0 to track the teal point on the back of the goat (and some of the other tracks), I am concerned that there are systematic annotation errors related to two axes: deformable and insufficiently distinct. 

To resolve this major weakness (although the TAP authors stay away from semantic annotations), I would similarly (as in 5) recommend a semantic analysis of annotation quality. If it is the case that systematically points that are relatively pronounced within deformable/articulated objects are wrong, then it really hinders the value of this dataset for tracking. The authors do already analyze their annotator's quality on the Kubric MOVi-E dataset, I am unclear on the quantity of deformable/articulated objects that were present in this analysis (if any) and believe an analysis of point-track annotation accuracy conditioned on the semantics of objects would be immensely valuable.  
correctness: The submission is technically sound, though the annotation quality could use improvement. Claims are supported, as the collected tracks improve TAP-Net performance. The methods used are appropriate and the paper is a complete work, although the centrality of TAP-Net is a bit strange for a dataset submission.
clarity: The submission is clearly written and well-organized. When including the supplemental material it seems sufficient for one to replicate the results.
relation to prior work: The tasks and methods are similar to existing methodology. Keypoint tracking and optical flow estimation have a long, rich history. The provided related works accurately describe the similar research in the area.

Beyond the included related work, other works are slightly relevant. “Learning to Estimate Hidden Motions with Global Motion Aggregation” already attempts to improve upon occlusion in RAFT, and could be used as a replacement optical flow driver and baseline here. Further, recent works like “Deformable Sprites for Unsupervised Video Decomposition” accomplish similar correspondence tasks, but done with sprites instead of points.
additional feedback: My primary concern is that "tracking any point" (a good goal) ends up turning into more of "watch all the failures in annotating hard to track points".  I like the concept but I'm worried about the execution. Any new data and annotations are good, but if new annotations are sufficiently noisy and low quality then no one will use them and they could harm tracking methods by providing false correct/incorrect signal.
review_title: "tracking any point" (a good goal) turns into "watch all the failures in annotating hard to track points on deformable/articulated objects"

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper constructs a new dataset for arbitrary physical point tracking. The proposed task, i.e., tracking any points (TAP), can benefit the development of embodied agents. Although optical flow-based methods can perform well in short-term videos for arbitrary point tracking, the drift accumulation problem hinders their application in long-term scenarios. A novel baseline method TAP-Net is proposed to deal with the problem and gets superior performance on the proposed TAP-Vid series point tracking datasets.
strengths: 1. This paper constructs the first dataset for long-term arbitrary point tracking tasks. Such a dataset can contribute to the development of embodies agents.
2. Thorough comparison between the proposed TAP dataset and previous datasets for point tracking has been made in the paper. It helps a lot in figuring out the difference and contribution of the paper.
3. Comprehensive statistical information of the proposed dataset and the detailed annotation process has been provided.
weaknesses: 1. Is there something like the error bar of the annotation provided? As demonstrated in the supplementary videos, some drifts or errors in the annotated points seem to exist, especially for non-rigid objects like the camel or black spawn. The drift or point disappearing happens even without occlusion. 

2. Is the proposed TAP-Net trained on the combination of those sub-datasets or individually trained and tested on each sub-dataset? Suppose the model is individually trained on each sub-dataset. In that case, I am wondering about the generalization ability of the TAP-Net on the unseen sub-datasets. Such experiments can help better benchmark the model's generalization ability on unseen scenarios and make the dataset more important.

correctness: Yes.
clarity: Yes.
relation to prior work: Yes.
additional feedback: Please see the Limitations part. I am looking forward to the authors' response and will increase my ratings if my concerns are well addressed.
review_title: A novel dataset for point tracking.

final_score: rating: 5: Marginally below acceptance threshold
review_content: summary and contributions: This paper formulates the problem of “tracking any point” on video data. This formulation differs from related problems by aiming to track points starting from any pixel in any frame, not using a variation of keypoints to limit point selection. Additionally, predicting occlusion of the keypoints is used to avoid the complications of attempting to localize obscured points.

The dataset is comprised of four sources of data, two existing real world video datasets and two existing synthetic video datasets. Real world datasets are constructed by crowdsourcing bounding boxes and rough point tracking. More detailed point tracking is approximated with a novel use of optical flow estimates that are iteratively finetuned by annotators. Synthetic datasets utilize the simulators for ground-truth point tracks. The quality of crowdsourced annotations is measured by crowdsourcing annotations of this synthetic data and comparing point tracks.

Several existing algorithms are modified for use as baselines in the TAP task. An additional deep learning algorithm, TAP-Net, is proposed and found to outperform the baselines on all datasets in terms of occlusion accuracy, average thresholded position accuracy, and average Jaccard thresholded position accuracy.

strengths: S1: TAP is a new, well formulated task that is clearly differentiated from tasks such as SFM, semantic keypoint tracking, and keypoint discovery.

S2: The track assist algorithm is novel, well evaluated, and useful for reducing annotation cost.

S3: The dataset collection process and content of the dataset is clearly described.
weaknesses: W1: How the dataset is utilized for evaluations is not clearly defined. Only TAP-Vid-Kubric has any mentions of a test split or the split’s size. None of the details in checklist item 3b are provided. Baseline methods lack any of the details described in checklist item 3d.

W2: The averaging of position accuracy metrics is not well motivated, with actual applications of these models likely to require accuracy at a domain-dependent fixed threshold. Averaging obscures whether there are specific thresholds where baseline models outperform TAP-Net.

W3: Discussion of impacts of the work are limited to technical impacts, with no discussion of positive or negative social impacts aside from the work of the Kinetics authors. **Clearly the NeurIPS ethics review guidelines were not adequately considered.**

W4: Use of existing assets is glossed over, not adequately answering checklist items 4b, 4d, 4e.

W5: Inadequate description of crowdsourcing participant compensation and potential risks presented to or found by Deepmind’s internal ethics committee.

correctness: The dataset is constructed in a sound way. The evaluation of human annotations is clear and supports the claims made. The dataset statistics presented in Section 5.1 were a bit ambiguous. It appears that these statistics should more clearly be stated to only apply to TAP-Vid-Kinetics, as the first sentence suggests that the following statistics apply to the entire dataset.

As described in W1, the evaluation methods are ambiguous. It is unclear how training, validation, and test sets are created for each experiment. Model hyperparameters and tuning methodology are not described. Compute time and resources used for baseline model training appear to be missing.

The ambiguity of the evaluation methodology causes a lack of context for the results. As described in W2, the metrics also provide little actionable insight, aside from occlusion accuracy. An average across all delta values would provide a better measure of performance independent of threshold choice. Evaluations at specific thresholds would provide more actionable insight, even if only summarized in the main paper with tables relegated to the supplement.

clarity: Writing is overall clear and concise. Figures and tables are well-made and unambiguous. However, there are notable absences in what was chosen to write about (W1, W3, W4, W5).
relation to prior work: The TAP task is related to prior tasks such as SFM, keypoint tracking, and keypoint discovery. TAP differentiates itself with explicit modeling of occlusion and no restrictions on the points that should be able to be tracked.

No previous dataset provides long-term tracks, precise points, class agnosticism, and non-rigid objects at the same time. Previous works have utilized algorithms to assist in label generation in segmentation and object tracking, but none have been utilized for point tracking.

The models benchmarked on the dataset seem to be straightforward extensions of existing techniques.

additional feedback: I would strongly recommend adding justifications to the answers given in the author checklist.
review_title: Novel Dataset with Flawed Execution

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: summary and contributions: This paper introduces a new benchmark dataset for the task of tracking any point (TAP), i.e., tracking an arbitrary 2D point defined at the first frame over the entire video. The dataset is composed of a mixture of real and synthetic data where for real data, human manually annotates the ground truth for a number of 2D salient points, and for synthetic data, perfect ground truth labels are provided. Analysis and statistics on the dataset, a new baseline model which outperforms existing methdos, and a comparison with existing datasets are provided.

strengths: [Strongly motivated, practical, and novel task definition] While robust long-term tracking of any points from a video is an important problem for robotics and 3D vision tasks, there exists no notable benchmark specialized for this task. This task might be able to open a new video tracking task beyond visual tracking and video object segmentation. 

[Clarity in writing and strong completeness] The writing is clear. The task is well motivated. The statistics and analysis on the dataset are compact, informative, and well summarized. Every claim is logically well justified. The process to annotate the labels is detailed, convincing, and practical. The suggested baseline method is reasonable, and cross-dataset evaluation highlights the importance of the new dataset.

weaknesses: For the final touch, it would be nice if the authors can provide an evaluation code, and explicit train/test/validation split.

correctness: Overall, it is sound. However, the authors should be able to clarify the difference between TAP and matching (not optical flow). 
clarity: Clear.
relation to prior work: Clear.
additional feedback: -
review_title: Review on TAP-Vid

Meta-review: metareview: TAP-Vid presents a benchmark that will be highly useful for tracking research for tracking arbitrary physical points on surfaces over long video clips. The reviews are all positive, with one reviewer raising ethical aspects in preparing the benchmark.  I find the rebuttal sufficient and adequate and hence recommend acceptance of the paper.
confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept