Venue: NeurIPS 2022 Track Datasets and Benchmarks

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The paper presents a very interesting dataset of Euclidean shapes and objects. It shows the limitations of current vision models in understanding these shapes and geometrical structures. This task is especially important since humans are good at it and current vision models are not. The paper is very well written and it is easy to understand the task and the motivation. The authors also introduce a dataset and a library to generate new datasets using the introduced geometric primitives.
strengths: 1. It introduces a very fundamental task in image understanding. Geometric shapes appear everywhere, and having the ability to generate complex shapes easily will be very useful for the community.

2. Employs novel near/far categorizations to comprehend the nuances of geometry. This will enable debugging future models and architectures for geometric patterns and help build models that are robust to geometric object perturbations like rotations. This will also enable research in disentanglement to understand specific geometric patterns that models learn and patterns that models struggle with.

EDIT: The authors addressed nearly all of the concerns and performed additional experiments to validate the claims. This dataset would be highly beneficial to the community and I have updated the ratings.
weaknesses: 1. The motivation to just use models trained on natural images is somewhat unclear. Deep learning models are domain specific and they often don’t generalize well to other domains outside the training data. For example, Berry et al. (1) changed the background of images, and that reduced the performance of the models. So it would be great to have a clearer description of why the authors think that these vision models should understand geometric objects. Thus, is this an appropriate goal to test the few-shot generalization of existing vision models trained on natural images?

2. Using the low-level features from pre-trained models to evaluate if models do learn some low-level features was an interesting experiment, but I would have loved some comparison and analysis on this part. Do vision transformers perform better than resnet models at low-level feature understanding with respect to geometric objects? Are some geometric structures too complex for these low-level features, like two circles and one line? What is a simple structure (like a circle or a line) for the model to generalize? This would also help illustrate newer insights into low-level features learned by these models. Since the vision transformers worked quite well compared to the other models, the low-level feature comparisons would be very insightful.

3. The model does near-far categorization. But there isn’t much analysis or insight into this category. What was the conclusion of the learned manifold? How robust are the learned images to near-versus-far-categorizations to simple perturbations in geometry (like angle distortions or scale)? In other words, do these models actually learn the near and far categorizations well from the geometric perspective?
correctness: I think the authors could make a stronger case by showing that training a model specific to these geometric images and evaluating if models do learn these geometric concepts. If the goal is to understand geometry why not build a model only to understand this concept? The few-shot reasoning to understand geometry by using vision models is not very intuitive to me because models do not generalize well to out of domain images as mentioned above (1). Even training on something like Google Draw dataset (https://github.com/googlecreativelab/quickdraw-dataset) could provide some interesting insights. Natural images seem very different from the domain of geometric objects to be few-shot generalizable.
clarity: The paper is well-written and easy to follow.
relation to prior work: Disentanglement literature will greatly benefit from building geometric objects and then understanding the nuances of interpolations of geometric objects. Most likely this wasn’t the author’s motivation but just wanted to highlight this information here.

Locatello, Francesco, et al. "Challenging common assumptions in the unsupervised learning of disentangled representations." international conference on machine learning. PMLR, 2019.
additional feedback: I think the authors could make a stronger case by showing that training a model specific to these geometric images and evaluating if models do learn these geometric concepts. If the goal is to understand geometry why not build a model only to understand this concept? 

The few-shot generalization using a dataset like google draw could provide useful insights into the few-shot generalizability to geometric objects.
review_title: Useful dataset with lots of potential applications

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This paper proposes a domain-specific language for the Euclidean concept realization and implemented rendering library, as well as two datasets for the few-shot concept learning task, which is used to evaluate the generalization capabilities of humans and pretrained deep learning models. The rendering library and datasets are good contributions to the plane geometry problems in computer vision. The experiments conducted show the gap in performance between humans and deep learning models, which is a good start for follow-up explorations to fix this gap.
strengths: 1. This paper conducted solid works on a less explored problem, which is related to the basic challenges and limitations of existing computer vision models. 

2. The developed library can be used in various follow-up works. 

3. The gained insights like the gap between humans and computer vision models, the difference in performance between CNN-based models and ViT are inspiring.
weaknesses: For it is finite in my knowledge, I think it can be problematic to compare humans and vision models using the task and setting in the paper, as well as using the few-shot generalization task to evaluate the human-like visual competencies of vision models. The reasons are as follows:
1. The humans actually conduct perception and reasoning (with Euclidean geometry knowledge and intuition)  processes while the vision models only conduct the perception (with representations from pretrained models) process. One possible supplementary can be adding experiments of humans with basic or no Euclidean geometry knowledge ( like elementary school students or infants).

2. If the target of the proposed benchmark is to evaluate the capability in Euclidean geometry concept learning or sensitivity to Euclidean geometry of different vision models, I think there can be better evaluating tasks rather than few-shot generalization task, for example, involving NLP models to discriminate whehter the concept and rendered image are corresponding or not, which can be trained on a generated large-scale dataset, rather than no training or fine-tuning. If there are follow-up research works on it, I think Capsule Networks can be a competitor on this benchmark.

3. The threshold is an adaptive value. I think it can be more reasonable to add the results by using the same models with initialized parameters, to compare with the results by pretrained models.
correctness: It is constructed in a sound way.
clarity: Yes. It is well written.
relation to prior work: Yes. It is clearly discussed how this work differs from previous contributions.
additional feedback: No.
review_title: Good contribution towards the vision problems involving plane geometry 

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: The authors propose a framework for Euclidean geometry: a language for representing anything which can be drawn with a straight edge and a compass.  They then show that humans are able to generalize such concepts after few examples, while state-of-the-art computer vision models perform considerably less well.  The idea is interesting, the gap between humans and AI models convincing, while the framework seems apt for benchmarking attempts to close this gap.  I only have a few minor comments

- p. 4, line 106: "we rejection sample realizations" -> "we reject sample realizations"

- p. 6, I think I understand what is trying to be conveyed here, however the paragraph starting on line 197 is a bit confusing with "each of 10 test images of 5 positive examples and 5 negative examples from either Close or Far..." when the paragraph above mentions 15 test images of 5 positive examples, 5 negative examples of Close, and 5 negative examples of Far

- p. 9, line 309: "negative society impact" -> "negative societal impact"

strengths: The idea is interesting, the gap between humans and AI models convincing, while the framework seems apt for benchmarking attempts to close this gap.
weaknesses: By giving more (than just 5) examples to the AI models, an exploration of how much the gap could be closed between humans and AI models would be interesting, but it is not necessary

correctness: Everything seems correct.
clarity: The paper is well-written.
relation to prior work: This is not my area of research, but the authors seem to cite and discuss prior work.
additional feedback: N/A
review_title: Interesting idea and apt framework

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: Euclidean geometry is an old mathematical branch. Even though those shapes do not exits in the real world humans are good in recognising those patterns. The paper tries to answer if machine learning models can be equally good at it. To answer the question the paper introduces a euclidean geometry shape dataset as well as a language to create those shapes. Also a novel few-shot generalisation benchmark is introduced to evaluate computer vision models on this dataset.

strengths: Comparison of human accuracy with computer vision accuracy to lay a suitable baseline
Evaluation of the Euclidean geometry with computer vision models
Introduction of a new language, which can render various geometric pattern, which then makes the dataset suitable for Few shot learning task, as well as extending it for educational purposes and maybe extension into n-dimensions
weaknesses: Some qualitative examples of the experiments would have been nice as well as bigger fonts in the figures, which makes the figures hard to read, when zooming in.

correctness: To my knowledge the methods seem to be correct

clarity: The paper was written clearly, as I could understand and follow it.

relation to prior work: All related work with geometric datasets and few shot learning have been mentioned.

additional feedback: I don't have anything to add here, except what was mentioned in the weaknesses.
review_title: Euclidean database for image generation

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The authors present the following:
- Two datasets and a benchmark task that test the generalization capacity of human and vision model on the Euclidean geometry concept. 
- Language to generate versions of the presented dataset flexibly. 
- Evaluation of human and pretrained model performances on the datasets 
- The benchmark task is a few-shot generalization of positive images from specific concepts (e.g., a circle inside a triangle).
strengths: - The language to generate samples from the dataset is particularly useful to further extend this research direction
- The author poses an interesting question through the benchmark task: can a vision model trained on natural images perform well on few-shot generalization of geometric euclidean concept? The comparison with human performance further strengthens the importance of this task.
- The paper is very well-written. A reader with minimal geometry concept learning background should easily get the gist of the paper.

weaknesses: 1.  Although the author clearly shows this task's importance, the reviewer still questions whether evaluating the pre-trained model for this task is fair. Perhaps this point can be clarified by citing literature that shows a strong relation between geometric concepts and downstream tasks.
2. It is unclear what are low-level and high-level features? Adding one-liner description of these will be useful.
3. What is the y-axis of the histogram for signifies? Is it the percentage of tasks with x% accuracy? Perhaps clarifying these will be useful, since the histograms are the main visualizations in this paper.
correctness: Dataset: Claims are correct and the dataset is constructed in in a sound way.
Benchmark: The paper is of sound evaluation methods and experiment design. Although adding citation mentioned in the 1st point in the weakness, or maybe adding another evaluation to prove that can further strengthen this paper.
clarity: Yes. But please add description of what is high level and low lever features.
relation to prior work: Yes.
additional feedback: Typo in line 106. "we rejection sample realization" should be "we reject sample realization"
review_title: Novel Dataset and Benchmark that exposes the gap between human and pretrained vision performance

Meta-review: metareview: This paper introduces a new dataset to evaluate the discrimination of geometric shapes. Using these images, the paper shows that humans (who probably have had extensive training with geometry) seem to generalize better across geometric shapes than standard computer vision algorithms which have had no direct exposure to training on geometric shapes but have been trained with natural images. 
confidence: 5: The area chair is absolutely certain
Decision: Accept