Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round1

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: summary and contributions: This paper describes a dataset acquisition and annotation protocol for a waste recycling task.  Images are obtained from a real-world industrial waste setup wherein waste material (including paper, brown paper, cardboard, soft / rigid plastic ) are imaged as they arrive in a conveyer belt using a camera setup.  Several datasets are provided to address fully supervised segmentation, semi-supervised segmentation, weakly supervised segmentation of object classes and binary classification of frames before and after material sorting tasks.   The contribution of the dataset is that it fills the void in realistic datasets for waste sorting task.  In addition, well known algorithms are used to demonstrate baseline performance and to illustrate that there is a significant need for performance improvement.
strengths: -- Real data (in the wild) for waste sorting task
-- Dataset will be of interest to machine vision and robotics community
-- Accessibility is open-source for non-commercial use
-- Addresses an important application area 

weaknesses: While I appreciate the efforts in the dataset construction for this important problem, there are a few points that are not very clear:

The image data is complex enough with variability in clutter, material types, occlusion. However, given that the problem involves material classification, isn't material classification usually done better by using hyperspectral imaging, use of polarization and other special sensors?  The authors mention the acquisition using NIR camera but as far as I can tell the benchmark data is only done on RGB data.   The authors claim that the datasets can be used for robotic grasping applications but I can't follow how the dataset in its present form is useful to robotics practitioners.  While I see potential extensions to the data collection process that may facilitate robotics research the current statement is an overclaim.

The sampling strategy used for the images is not clear - no mention of the conveyor belt speed is made to get an understanding of the degree of redundancy between subsequent frames in the video.  

The authors claim that this is large scale dataset. While the number of images is in the order of 1000's, the complexity of the application may necessitate sampling across several hours of operation, with diverse variations in waste.  It is not clear how the current dataset is sufficient for the application.
correctness: See comments above on weakness. 
The evaluation methods and experiment design is at a baseline level preliminary and could be strengthened.  Its hard to judge from the figures in the supplemental (e.g. figures 9, 12) what really the current baselines are successful at. There is no discussion of the specifics of failures.  I would recommend iterating and improving the baseline to provide adequate insights into the achievable performance using state of the art techniques.
clarity: The paper is well written although the paper is redundant (there is significant overlap between the main body text and the answers for the checklist).  
relation to prior work: Adequate discussion of past datasets relevant to this applicaton is presented and there are statements on how their dataset differs from the previous datasets. 
additional feedback: Can you comment about the points discussed in the weakness section?   The main concern I have is that the dataset may be more of academic interest given that real-world machine vision systems will employ alternative strategies for classification of material types.  Moreover, can you comment about how this dataset helps the robotics researchers?
review_title: An annotated dataset for waste sorting with baselines for segmentation and detection task is constructed. 

final_score: rating: 5: Marginally below acceptance threshold
review_content: summary and contributions: This paper presents a in-the-wild industrial-grade waste detection and segmentation dataset called ZeroWaste, which can serve for fully, weakly and semi-supervised detection and segmentation tasks.

The authors present baselines for fully, semi and weakly-supervised segmentation methods.
strengths: The proposed dataset is meaningful for more efficient waste recycling, quality control of the sorting process and establishing safer work conditions for the MRF workers.
weaknesses: The novelty of this paper is limited, since it only conducts experiments with current existing state-of-the-art approaches.
correctness: Yes. The dataset is constructed in a sound way.
clarity: Yes. The paper is well written and easy to follow.
relation to prior work: Yes.
additional feedback: -
review_title: Waste detection and segmentation dataset for automated waste recycling

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: The paper proposed 3 datasets including ZeroWaste-f dataset that is fully-annotated for industrial waste object detection; unlabeled ZeroWaste-s set for semi-supervised learning, and ZeroWaste-w dataset for binary classification. Several baseline object detection and semantic segmentation algorithms are evaluated on the proposed datasets. 
strengths: While the dataset has lots of environmental meaning, the authors also provide comprehensive evaluation on different baseline algorithms

 


weaknesses: The dataset is a good initiative, however, the task of object detection from MRF is very hard. From the images, we can see there are lots of occlusions, partial objects, noisy background, uneven lighting conditions etc. And the labels only cover the complete objects, which reduced the usability of the dataset. And the big gaps between training and test are very concerning.
correctness: yes
clarity: yes
relation to prior work: yes
additional feedback: no
review_title: A good initiative for waste recycling

Meta-review: Reviewers agree on the importances of this dataset. However, a few more iterations may be needed for this to be ready for publications - we recommend authors consider submitting the second round with these revisions. To mention a few, 1) clarifying how “other ways” to do the same task (e.g., NIR data) is relevant, pros and cons 2) clarify how the proposed applications are relevant e.g., robotic grasping 2) [important] discuss failure modes to provide insights 4) other computational aspects: iterate and improve baselines, clarify sampling strategy etc.

Decision: Reject