Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round2

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper presents ClevrTex, a textured benchmark for Unsupervised Multi-Object Segmentation. It is synthesized using similar procedures as Clevr, but differs in that it adopts a textured background and textured objects. Its goal is to test the current arts' ability to work on textured objects instead of textureless objects, and experimentally shows that texture poses a huge challenge to existing arts. Extensive studies are performed on a varieties of dataset variants, showing the necessity of this dataset as a challenging benchmark to evaluate Unsupervised Multi-Object Segmentation.
strengths: + The dataset is well motivated and well designed. Its utility is demonstrated through a carefully designed set of experiments. The experiments cover all aspects of dataset variants one can possibly think of.
weaknesses: - It might be better to provide some (high-level) insights into why texture poses a challenge to state-of-the-art methods
correctness: - Correct.
clarity: - Very well written.
relation to prior work: - Clearly discussed.
additional feedback: - See comments above.

============================= 
Final comments: I am happy with the answers from authors. I keep my original rating.
review_title: A Good Benchmark for Unsupervised Multi-Object Segmentation

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The paper presents a synthetic dataset with a similar make up to Clevr (Johnson et al 2017), but visually much more challenging, with diverse lighting, textures and object shape perturbations. The principal motivation for the benchmark, is creating a more challenging test ground for unsupervised scene understanding and decomposition methods, and the paper presents extensive analysis that shows that the variation in textures causes existing methods' performance to degrade drastically. The dataset is well-motivated and will be a useful contribution to enable research in this area. 
strengths: The dataset is well-motivated and is a timely contribution to spur on new research in unsupervised scene understanding. Extensive analysis of existing methods when trained on this data and tested on several variants is convincing and a valuable contribution as well. Additional ground truth data can also make the dataset useful beyond the applications described as the primary motivation.
weaknesses: The dataset construction pipeline should be described more precisely. Specifically:

- L 120: How were the photo backdrops selected, how many were used, and what were the splits for train/test?
- Section 3.1: range of the number of objects per scene; camera parameters, scales. 
- Section 3.2: more details on material splits.
- Tb 1: it is not clear what the total time is for, is it training, inference, what is the input...?
- Supplemental: include all 60 materials used; include lighting variation demonstration; include object shape perturbation examples. 

correctness: The construction and analysis seem sound.
clarity: The paper is clear with the exception of a few missing details. The abstract should list important details about the dataset, including number of images, number of materials, variations, as well as additional ground truth available -- this will make it easier for readers to asses if the data is suitable for their purpose.
relation to prior work: The key difference is clear. However, this should be more clearly presented. A table summarizing differences with Clevr and any other relevant datasets should be included for clarity, including number of images, number of backdrops, number of object classes, number of materials, range for the number of objects per scene, ground truth annotations. 
additional feedback: This is useful work that will benefit the community.
review_title: Useful Benchmark, minor details missing

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This paper presents a new texture-rich dataset that can be used for object level 3D understanding such as segmentation. Compared to existing 2D and 3D datasets, this one contains higher visual complexity and is hard for current methods to perform very well on it.
strengths: This paper clearly verifies some of my thoughts in the last few years, that is, many of existing computer vision systems failed to consider visual realism and complexity, especially those that heavily rely on synthetic data. As a computer graphics researcher, I fully understand the importance of having a texture and material rich dataset, therefore I appreciate any effort along this direction.

The variants subsets look interesting to me since I can see authors carefully construct them to test specific capability of existing methods, and I think this is the correct way of decomposing the dataset into more useful parts. 
weaknesses: Although I do like the idea of increasing the visual complexity of multi-object dataset, there are some weakness of this work:
1. The dataset diversity is still questionable in terms of material, texture, shape, and lighting. Currently there are only 60 materials, a few base shapes and texture maps, and a few different lighting conditions used to create the dataset. I would recommend using Adobe's material library which contains much more realistic ones. It is always a good idea to apply some levels of bumping on the surface to further increase the variation. As for lighting, I also recommend using some environment light datasets that are currently available to enrich the random area lights.
2. There is no new methods proposed in this work to address some challenges of the new dataset, so the technical contribution is another weak point of this paper.
3. The paper is written in a way that makes me think it is only useful for object segmentation. However when I took a look at the dataset there are more auxiliary channels such as albedo, depth, normal, etc. So this looks to me can be also used for other tasks such as depth or orientation estimation for robotics tasks. Only considering a single task however weakens the usefulness of the dataset.
correctness: I believe the contents are correct, since both dataset and code are available online to allow anyone to check.
clarity: I believe so.
relation to prior work: I think the authors did a good job on discussing about previous works and datasets by showing some pictures that contain noticeable differences.
additional feedback: My suggestions for improvement are included in the weakness comment.
review_title: Interesting Textured Dataset

Meta-review: All reviewers recommend acceptance and point out the good execution and potential impact of the work in the community. The AC has no reason to overturn the reviewers' decision. Congratulations to authors!
Decision: Accept