Venue: NeurIPS 2022 Track Datasets and Benchmarks

Reviews:

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: The submission introduces new pixel level and segment level semantic annotations on top of the existing EPIC-KITCHENS-100 videos. Annotations include temporal actions over short and longer time spans (e.g. flour --> dough), pixel level segmentation of active objects, relationships between hands and active objects and a new type of annotation indicating where a given object came from. Some statistics about the data are provided.

Data is annotated via a novel proposed pipeline, done in three steps: (i) identifying active objects, (ii) pixel based annotations on the selected objects, helped by an AI-powered annotation tool, and (iii) object-hand relations. These sparse annotations are densified later with the help of models and the dense annotations (not manual) are also provided.

The submission also proposes benchmarks on three tasks: video object segmentation, hand-object segmentation and relations with objects, and a novel "where did this come from" task. Model baselines are provided for the two first tasks and an oracle-based baseline is provided for the last task.

The data is released in a clear format and is easy to use.
strengths: - The dataset provides annotations on transformations (e.g. flour --> dough), which is an interesting concept still to be understood well by vision models
- The dataset provides short and long temporal actions
- The amount of annotated frames is great
- The proposed set of tasks in the benchmark has both more traditional tasks and novel attempts of making vision models able to do human-like tasks (e.g. WDTCF)
- Building on top of Epic Kitchens enriches the set of annotations and contributes to broader research
- Data is easily accessible in the format provided
- Different strategies to improve annotation quality were used

In summary, the dataset is well built, accessible and definitely contributes to the broader research community by both providing more data for existing tasks and also proposing new tasks that can push the community towards solving perception.
weaknesses: - I would have liked to be provided with code to visualise annotations rather than just images in the submission
- Statistics on annotation errors would be a plus (e.g. what proportion of manually verified annotations needed correction?)
- Reporting more diversity metrics on the annotators would help
correctness: I didn't find any incorrect claim.
clarity: The submission is well written and clear. No concerns on that.
relation to prior work: Clearly discussed.
additional feedback: No additional feedback
review_title: Review for Epic-Kitchens VISOR

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The paper proposes VISOR benchmark, a dataset with segmented kitchen videos along with three tasks: semi-supervised video object segmentation, hand-object segmentation relations, "where did this come from" back in time segment tracking. Tasks are accompanied with metrics and baseline models. Data is under permissive license CC BY-NC 4.0.
strengths: 1. Data labelling pipepline is described in detail. That may be beneficial for other researchers.
2. Well-known methods are used for baselines (STM and PointRend) along with standard metrics.
3. Permissive license and no ethical concerns.
weaknesses: 1. While benchmark labelling, tasks and baselines are well written, motivation for creating benchmark is described poorly. Why these three challenges are selected? What kitchen applications are possible if a method shows good quality on one of the tasks?
2. Introduction compares VISOR with previous work very briefly, I believe that adding separate Related work section would benefit the paper.
correctness: All claims made in the paper are correct. Dataset construction is sound.
clarity: The paper is well written and has extensive description of all benchmark details.
relation to prior work: Relation to prior work is briefly discussed in introduction.
additional feedback: No additional feedback.
review_title: A video segmentation benchmark with kitchen scenes

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper introduced a richly annotated dataset for video object segmentation.  Compared with previous datasets, the new VISOR dataset has more frames, more annotated masks, and more action and entity classes.  The authors also defined a new task, Where Did This Come From (WDTCF), to show that the annotated long-term videos may enable more complex reasoning problems.  I see both the dataset and the task as major contributions.
strengths: This is a strong submission.  A richly annotated video dataset is always welcome.  The annotations seem to be of very high quality.  In addition, the authors have introduced an interesting new video reasoning problem.
weaknesses: I actually don't see many weaknesses in this submission.  I have two minor comments.

1. The discussion of related work is a bit thin, in particular with [33].  It would be great to better explain and motivate why VISOR is needed, and what problems it may enable us to solve with e.g., long-term videos?  The WDTCF task is a nice demo, but it can be motivated to highlight why such a task is needed.

2. The authors have not provided any quantitative evaluation of annotation quality.  From the demos, the annotation quality seems very high. Just to make the dataset contribution more solid, I wonder if there is a way that this can be systematically calibrated (e.g. wrt with some ground truth from mocap?)

I'd be happy to increase my score if the authors may address my concerns, especially [2].
correctness: Yes, the claims are well supported. 
clarity: Yes.  The writing is clear.
relation to prior work: The discussion is a bit short, but mostly fine. However, I'd like to hear more about how the dataset differs from UVO [33].  For example, is it mostly 12s vs. 3s on the average sequence length?
additional feedback: None.
review_title: Useful dataset, richly annotated

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper extends EPIC-KITCHENS dataset for pixel level annotation.  In addition to both hands, active entity is pixel level annotated forming large scale pixel and action level annotation. The annotation was conducted on sparse frames and dense frame data is interpolated from neighboring sparse frames. This benchmark can be used for three tasks VOS, HOS and WDTCF.
strengths: The dataset is relatively larger than other datasets of the same style.
The dataset is built with proper use of AI-powered segmentation tool. 
Solid and thorough inspection is performed for the dataset.
weaknesses: I don't see notable weakness in this paper.
correctness: The construction procedure of the dataset seems to be sound enough.
clarity: The paper is clearly written.
relation to prior work: I don't have expertise in this field.
additional feedback: I don't have additional feedback.
review_title: large scale dataset with action and pixel level annotation

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This study is dedicated to a new semantic segmentation benchmark for segmenting and tracking hands and active objects in ego-centric videos, called VISOR. It is created on the top of EPIC-KITCHENS video dataset, yet introduces new annotations and three completely new challenges: video object segmentation (VOS), interaction understanding (Hand Object Segmentation, HOS), and long-term reasoning (Where Did This Come From? or WDTCF). The labeling is performed semi-automatically: the semantic masks are labeled via a AI-powered tool providing user guidance. Moreover, trainable STM methods are leveraged to extrapolate manual pixel-level semantic segmentation masks to the neighbor frames, giving dense video annotations.
strengths: Good paper overall.

1) A large amount of work has been obviously done, that includes developing labeling tools, creating verification procedures, formulating object categories, designing challenges, and setting up baselines. 

2) The challenges are fun and inventive. Despite I have some doubts on the third one, I find the first two quite meaningful.

3) The appendix provides a detailed and comprehensive description of all procedures performed while creating the benchmark. I personally appreciate a rational and thoughtful approach to frame selection and labeling based on a common sense and supported with the statistical evidences.

4) This benchmark has a practical value: it could be used to facilitate creation of an AI-powered home assistant providing smart guidance during a meal preparation (e.g., if connected with a camera facing downwards placed underneath a cupboard).
weaknesses: 1) Well, I personally dealed with the tasks implying recognising actions and objects in the kitchen. However, for a reader without such an experience, the motivation of creating this benchmark might be non-obvious. I would recommend to articulate possible usage scenarios and potential applications more clearly.

2) The test set contains 20% of the masks and 14% of the frames. Does this mean that the mask-per-frame distribution for the test set differs from the one for the remaining data? In other words, how can we be sure that the test subset is not biased w.r.t. the train subset?

3) WDTCF (Where Did This Come From) challenge seems to be fun, but a problem statement is a bit controversial to me. The fridge and jar represent different categories of containers that are not mutually exclusive and cannot be opposed in a real life (e.g., a jam might come from a jar taken out of the fridge, so various combinations are possible).

5) Among the annotated objects, there are ‘watch’ and ‘candle’. I am not really sure whether they are significant in the meal preparation context and should be annotated. 

6) The VISOR annotations include such an object as 'foot'. I am terribly curious about how a foot can be used while preparing a meal (not a weakness though, but I would be grateful for an answer).
correctness: The authors introduce three new challenges associated with the benchmark, two of which are completely new (HOS and WDTCF), so the novel evaluation protocols are designed. However, they seem quite reasonable, and rely on the standard well-known metrics (Jaccard IoU, Boundary F-measure, Mask AP).
clarity: The paper is clear and easy to follow.
relation to prior work: The relation to EPIC-KITCHEN dataset serving as a basis for VISOR is made clear. The authors also provide a quantitative comparison with the existing datasets featuring both pixel- and action-level annotations: overall, VISOR contains longer video sequences and provides more manually annotated masks on diverse actions and objects. Having less images than the largest benchmark named YTVOS, VISOR features much more masks. VISOR does not have the largest masks-per-frame value, but it is inferior only to the UVO 0.5 dataset, which is sparse. Moreover, it is superior to all other datasets in terms of average sequence length and total capturing time.
additional feedback: See Weaknesses and Documentation.
review_title: Novel benchmark of ego-centric videos of meal preparation

Meta-review: metareview: This paper introduces new annotations on top of the EPIC-Kitchen 100 videos, which contain pixel-level instance and semantic annotations on objects. On top of the valuable instance mask annotations, such as hand-object interaction. The annotation volume presented is significantly larger than prior works in multiple statistics. Based on the added annotations, the author also propose a new task, "Where Did This Come From (WDTCF)", to demonstrate the values of added annotations for more complex semantic tasks. This is an important area to the community, and the dataset enables both studies in important classical problems (e.g. VOS) and newly proposed tasks. 

All the reviewers have suggested acceptance without major concerns to the paper. Meanwhile, the authors of the paper considered the reviewers' suggestions and improved the paper to include:
1. More discussion / separate section on related works (t9qp, XaV1)
2. Visualization tools (sj6U)
3. Baseline code (sj6U)
4. Expanded discussion of the motivation for the work and proposed task (cefW)

I believe that the authors' responses and updated materials address most of the issues raised by reviewers. I think this benchmark will be a good contribution to the computer vision community.
confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept