Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round2

Reviews:

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: summary and contributions: This paper proposed datasets, a new evaluation metric, and baselines for video panoptic segmentation. I believe this work will promote the development of this field. Their contributions include:

- based on the KITTI dataset, the authors created KITTI-STEP and MOTChalleng-STEP datasets with pixel-level annotations for every frame.
- proposed a new metric to evaluate the task of video panoptic segmentation.
- introduced baselines and tested them on the created dataset to show the challenges.
strengths: detailed and easy-to-follow descriptions including the processes of data collection and annotation, metric computation and comparison, baseline implementation, and result analysis.
weaknesses: - Although the comparison of evaluation metrics is described to show the benefits of the proposed metric, I think the computational complexity of each metric is worth analyzing.
- Figure 3 is not easy to understand to me. Are the colors of gt cars correct? And the color masks may not necessary.
correctness: Yes
clarity: Yes, the paper is well written.
relation to prior work: Yes, it is clear.
additional feedback: Please see the weakness.
review_title: Very nice paper, I would like to give a high rating.

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper presents a new dataset and metric for segmentation and tracking. The old metrics for segmentation and tracking are carefully study and their limitations are explained. The new proposed metric takes into account the segmentation and tracking performance without preferring one over the other.
strengths: -	The paper is a pleasure to read and easy to follow.
-	The limitations of the previous data set and metrics are carefully studied and explained.
-	The design of the new STQ metric is very well explained.

weaknesses: -	In the main manuscript, there is limited information about how the data set was collected and more visual examples should be added.
correctness: The metric seems correct and it is backed by a lot of explanations.
clarity: The paper is very clear.
relation to prior work: The related work is carefully studied.
additional feedback: -	In the main manuscript, there is limited information about how the data set was collected and more visual examples should be added.
review_title: Review

final_score: rating: 5: Marginally below acceptance threshold
review_content: summary and contributions: This paper presents two datasets for long-term video panoptic segmentation. The datasets are annotated with high frame rates and long durations. In addition, the authors also propose the Segmentation and Tracking Quality (STQ) for better evaluating the performance of VPS models.
strengths: - The KITTI-STEP and MOTChallenge-STEP datasets are meaningful due to their high-frame and long-duration annotations.
- The challenges based on the proposed dataset are indeed contributions to the research of the related fields.
weaknesses: I appreciate the efforts by the authors to create the datasets and organize the challenges. However, I have several concerns about this paper.

- The task definition that "Tracking Every Pixel" is somehow overclaimed to me. I understand that the authors may mean that their evaluation considers every pixel. Yet, "Tracking Every Pixel" always reminds me tasks like optical/scene flows that define identities for each pixel. This paper just tracks at the instance level, and is exactly a video panoptic segmentation task. Hence, it is improper to overly emphasize "every pixel" in this paper.

- In the first item of  "Contributions", the authors claim that the proposed datasets are "more suitable" for Spatio-temporal dense tasks. This is not very convincing to me, as in the proposed dataset only two classes ("Pedestrian" and "Car") are considered as instances, which ignores many other traffic participants as defined in CItyscapes. Moreover, there also lacks analysis about the quality of the annotations.

- I also noticed that the STQ has been proposed in [a]. Though [a] is for 3D LiDAR data, the very similar formulation makes STQ hard to be a contribution of this work.

- The content of  Ln63-69 is overlapped with Ln46-62.

- It would be better to add some qualitative analysis about the datasets.

[a] 4D Panoptic LiDAR Segmentation, CVPR21

correctness: Yes
clarity: I believe that it is improper to overly emphasize "every pixel" in this paper, since
- All the previous evaluation metrics are computed from pixels.
- The proposed STQ actually measures at the instance level, and its advantages for pixel-level quality are not obvious.
relation to prior work: Yes
additional feedback: No
review_title: This work proposes new datasets and evaluation metric for the video panoptic segmentation task.

final_score: rating: 5: Marginally below acceptance threshold
review_content: summary and contributions: This paper introduces two datasets, KITTI-STEP and MOTChallenge-STEP, for the task of video panoptic (segmenting and tracking pixels in a video) by extending two existing datasets KITTI and MOTChallenge. To improve over existing benchmarks, this paper introduces extra annotations by labeling more semantic classes from the background. Compared with previous benchmarks, the proposed contain more dense and long-term annotations. Further, the authors also investigate the metrics for this task and present a new metric by providing an accurate and intuitive comparison against the GT at a fine-grained level. In the final part, experiments are conducted for some baselines with analysis.
strengths: This paper introduces two datasets for video panoptic segmentation by providing extra annotations for more semantic classes. I appreciate this effort.

The authors investigate several metrics and develop a new one for this task.

Experimental results are shown for a few baselines with analysis. This can provide comparisons for future research.
weaknesses: My biggest concern with this work is its major contribution, i.e., the two proposed benchmarks KITTI-STEP and MOTChallenge-STEP. There are some questions:

(1) how do these two benchmarks help improve the understanding of scenes in videos? In addition to adding some more annotation, I don't see much big difference from existing datasets. In addition, in these extra annotations, some are actually not necessary to track. 

(2) what do we learn from Fig. 1? I don't see any indication in the paper to Fig. 1. Is it the result of baseline? Or do you want to show the annotation in your datasets?

(3) From my understanding, Fig. 1 is actually similar to video semantic segmentation. Is there anything related to tracking each pixel in the video?

(4) The statement "tracking every pixel" is confusing (at least to me). When we talk about tracking something, we usually assign an identity to that object. For example, tracking single object means we need to assign a unique label to the target of interest. Likewise, tracking multiple objects needs to assign IDs to different objects. When I see tracking pixels, I will think about assigning IDs to each pixels regardless of their semantic categories. Correct me if this is a common way to call the task in your field.

(5) How do you plan to explore temporal information in the videos with average length 51 (KITTI-STEP) and 187 (MOTChallenge-STEP), especially considering there exists information redundancy in these videos?

correctness: The claims seem to be correct.
clarity: Yes.
relation to prior work: The authors provide rich discussions with previous works.
additional feedback: Please check out the weakness part.
review_title: Two datasets for video panoptic segmentation, but with less new insights introduced

Meta-review: The paper received somewhat diverging ratings, from strong accept to weak reject. The AC read the paper, the reviews and the authors' responses. I do understand the reason why some concerns were raised by reviewers about what it means to track every pixel and other such nuances, but I do find myself to side with the authors. The dataset is interesting and opens the door for more comprehensive and challenging visual recognition. The paper is well written and well executed. So, congratulations to authors!!
Decision: Accept