Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round1

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: summary and contributions: The paper presents a novel image dataset to facilitate 3D shape bias research in neural network communities. The dataset is built by rendering designed 3D shapes from different viewpoints, and the data are augmented in various ways. This dataset is used to study the shape bias of a class of 3D reconstruction models that only requires 2D supervision. 
strengths: + The motivation of the paper is to improve the ability to go from images to their underlying 3D shapes, which is a topic worthy of attention. A synthetic dataset is presented to validate this problem.
+ The data generation involves many factors including background texture, viewpoint change, image degradation, distortion, and so on.
+ The design of the experiment is very detailed. Based on the proposed data, the influence of three-dimensional information on the robustness of tasks such as classification is demonstrated.
weaknesses: - The shapes for building the dataset consist of only dozens of simple basic geometry, which is far from representing the daily structures.  The rendering settings are simple and the images are not photo-real.  These factors determine the huge difference between the generated data and the real data.
- The proposed classification task is too simple to be compared with the classification and recognition task at the level of ImageNet. It is doubtful whether the final test conclusion can be applied to the complex prediction model.
- In terms of a dataset, the scale is small and the composition is simple. I don't think the dataset at this level would be very helpful to the research community.  The article presents meaningful goals to study the role of shape bias in the robustness of vision models, but the scale of the dataset has limited the validation and makes the conclusion less convincing.
correctness: The correctness of this paper is limited to the experiments of the proposed small data sets and classification tasks, and the correctness of the experimental conclusions is difficult to be confirmed on other complex vision tasks.
clarity: The paper is generally well written but there is still room for improvement in language.
relation to prior work: It's a little strange to put the work after the experiment section. In the related works, the introduction of 3D datasets and synthesis based methods is too brief.
additional feedback: Considering the limited contribution as the dataset and benchmark, and the fact that the data was built to illustrate the solutions to improve the robustness of vision tasks, I think this paper is not suitable for evaluation on the datasets and benchmarks track. The diversity and rendering photo-reality of the image should be improved to enhance the credibility of the conclusions.
review_title: The dataset promises too much but the effectiveness is uncertain.

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary and contributions: This paper introduces an image dataset for facilitating 3D shape bias research. The dataset is aimed to exploit the relationship between 2D features and 3D shapes with small components of objects. Besides, it emphasizes the variation across shape features. The image generalized from 3D objects via different viewpoints can enlarge the 2D image dataset, which can better generalize to unseen rotation and shift than regular CNN.
strengths: - This paper build a 2D image dataset with 3D Geons from different viewpoints and background that can make traditional CNN more robust to transformations like rotation and shift. 
- This paper exploit Geons to generate the dataset which is simple yet realistic to obtain shape variations. 
weaknesses: - The dataset is built on Geons which is simple to generate a group of shape variations. And there are 10 Geon categories in the dataset. I am concerned whether these Geons can help better understanding categories in ImageNet for 2D and ShapeNet for 3D. Though these simple Geons can easily produce different variations based on latent features of Geons, there is no explicit connection between Geons and objects in real scenes. I'm not sure the dataset could facilitate the understanding of objects in real scenes and contribute to the communities. 
- In L151, authors prepare three versions of Geon3D-10: black, random textured and correlated background, to quantify the effect of textures. I can not figure out the meaning of this process since it just changes the background texture as shown in Fig. 3 (Right) while the texture of Geons is not change. In 2D image bias research [1], they change both background and object texture to generate variations. I think that here should consider the effect of textures for both objects and backgrounds can make experiments more convincing.
- I have downloaded they Geon3D dataset, I found that the image of each Geon is a projection from different viewpoints. RotationNet [2] provides a dataset that includes multi-view images of 3D objects on top of  ModelNet. I think this is a similar work and it also includes 3D shape bias. It is better to explain the superiorities of Geons than the whole object for 3D object understanding.
- The dataset lacks of detailed statistics and explicit validation for the correctness of data collection.
- The Geon3D benchmark is vague. There are no explicit metric definitions for evaluating the results on the proposed dataset. 

[1] R. Geirhos, P. Rubisch, C. Michaelis, M. Bethge, F. A. Wichmann, and W. Brendel. ImageNet- trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness. In International Conference on Learning Representations, Sept. 2018.

[2] Asako Kanezaki and Yasuyuki Matsushita and Yoshifumi Nishida. RotationNet: Joint Object Categorization and Pose Estimation Using Multiviews from Unsupervised Viewpoints. Proceedings of IEEE International Conference on Computer Vision and Pattern Recognition (CVPR). 2018.
correctness: The definition of the Geons is explicit, while the dataset lacks of detailed statistics and validation procedure. The benchmark does not have an explicit definition of evaluation metrics.
clarity: This paper is not well written. The proposed dataset and benchmark lack of some key components as mentioned above, which is not up to the demand of this track.
relation to prior work: This paper clearly discusses the differences from 3D datasets, e.g. ShapeNet, OASIS and Rel3D. RotationNet should be compared in the paper as mentioned in weaknesses.

additional feedback: It is interesting that study the Geons to simplify the 3D object understanding. However, the dataset may need to find the relationship between Geons and  complete objects to demonstrate its contributions, which is more meaningful for the community.

The dataset and benchmark should be formed in a more canonical way and include more detailed information. 

\* More feedbacks are in comments.
review_title: Geon3D is a novel dataset but not good enough.

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary and contributions: The paper introduces the dataset Geon3D, containing images of simple geometric blocks (Geons) generated from a set of high-level shape features. The dataset is motivated by studies on huma perception, and its goal is to study the role of 3D bias in visual recognition. The authors perform extensive experiments that show how networks pre-trained on 3D data improve the performance over those trained directly on 2D images. Specifically, networks with a differentiable neural rendering module show higher accuracy with background variations, better generalization for viewpoints unseen at training time, a higher degree of robustness against standard image corruptions and adversarial attacks.
strengths: + The dataset has a high degree of novelty up to my knowledge.
+ The experiments and analysis are, in my opinion, insightful. They show very clearly the benefits of 3D shape bias in a simple dataset, which is free from distractors appearing in large datasets.
weaknesses: - The paper is focused on the experiments and the analysis, which are relevant, interesting, while dataset-related aspects are more limited. Although not completely out of scope, I think the paper fits better other calls, to me the dataset is not the central part here, but the experiments: I cannot see a reason why 3D shape bias should be evaluated in the Geon3D dataset and not others, e.g., Kevin Lai's Washington RGB-D objects dataset.

- The simplicity of the generated data, while highlighting some interesting aspects, raises questions on the generalization of the conclusions to more realistic data. In my case, for example, I wonder if the benefits of 3D pre-training still holds for textured objects or complex shapes. The conclusions of the authors are very interesting but they do not explore the limits of their findings, that is, if the conclusions are still valid for more realistic setups. 

- The value of the dataset for future uses is unclear. In most of the tasks of the paper, the accuracy of the authors' models is already very high, so its value for developing new approaches might be limited. The authors suggest using it as sanity check, which is quite a limited use, and again, does not ensure that approaches will work in more complicated setups. The authors did not clearly define benchmarks or specific tasks, nor they detail the file structure of the data. Again, in my opinion, the paper is not data-centric and the focus is on the authors' experiments and analysis.
correctness: The paper is correct up to the extent I checked. The dataset is correctly motivated, and the experiments are clearly explained and correctly analysed.
clarity: I found the paper very clearly written.
relation to prior work: The paper references the most related works, up to my knowledge.
additional feedback: My opinion of this work is that the data is not its most relevant part, and the experimental analysis is insightful and promising. I think this track is not the best place for this work. And I also think that the work, although interesting, leaves open questions regarding more realistic images. I would encourage the authors to complete their work by extending the experiments (textured objects, more general backgrounds, real-image datasets) and submit to another venue.
review_title: Interesting analysis, limited dataset contribution

Meta-review: All reviewers recommended reject. Generally, the reviewers found the dataset to be too simplistic and synthetic and were not convinced about its utility. After reading the authors' responses, the AC agrees with the reviewers. 
Decision: Reject