Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round2

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: summary and contributions: This paper proposes a fine-grained church dataset for architectural style classification, sourced from Wikipedia. Some of the images are also annotated with bounding boxes of characteristic visual features. This dataset can be potentially used for different research problems,  fine-grained distinctions, imbalanced classification, and learning with a hierarchical organization of labels, where only some images are labeled at the most precise level. A classification baseline for this dataset is provided.
strengths: 1. Such a church dataset for fine-grained recognition does not exist before. This newly proposed dataset can benefit some specific fields such as fine-grained classification.
2. The dataset collection, cleaning, and annotation processes are described in detail and clearly.
3. The bounding box annotation for some images is interesting and useful for helping the model to learn fine-grained recognition.
weaknesses: 1. As the authors mentioned in the paper, there are label noise and bias in the dataset, which might affect fine-grained recognition training.
2. The dataset can be potentially used in many tasks, but the authors only provide a simple baseline for image classification with resnet-50. Having a more comprehensive baseline on different tasks and different previous methods would give the readers a better understanding and insights of the dataset. For example, a benchmark of the performance of previous fine-grained classification and imbalanced classification methods on this dataset, the effect of imagenet pretraining on this dataset, etc.
correctness: The claims made in the submission are correct.
clarity: The paper is well written.
relation to prior work: It is clearly discussed.
additional feedback: N/A.
review_title: A fine-grained church dataset

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: The authors introduce a novel dataset for architectural style classification, consisting of 9,485 images of church buildings, which can be used for fine-grained distinctions between classes based on subtle visual features, a highly imbalanced class distribution, a high variance of viewpoints.
strengths: 1). The proposed architectural styles dataset is novel and practical, which covers different types of images and labels compared to prior datasets. 
2) The paper provides detailed statistical information.
weaknesses: 1)  Limited takeaways. The authors claim that there are numerous real-world challenges: L456 fine-grained distinctions between classes based on subtle visual features, a highly imbalanced class distribution, but do not provide detailed experiment and insight to prove the related contribution to prior dataset or methods. Some things the authors could have included, are e.g. discussion of dataset difficulty (prior vs. theirs for fine-grained distinctions between classes tasks) as well as to show that the same model trained on the proposed large and the existing datasets. This is also something to do for the Imbalanced Class Distribution challenges. 
2) Contribution/Comparison to prior datasets. It remains unclear whether the proposed dataset is indeed more challenging. I know that there are few prior works for architectural styles, and it's not the authors’ fault. But what is the main difficulty for the current hierarchy classification, imbalance class, fine-grained distinctions between classes, and what is the main solution for these challenges? Whether these methods are effective for the proposed dataset?
3) Chosen methods. The authors only provide one simple baseline with ResNet-50. It's hard to convince for so simple experiment setting without new insight and analysis.
correctness: Lacking the convincing experiment, and the comparison to prior dataset or methods.
clarity: Paper is well written.
relation to prior work: No, there are many related previous works that need to discuss and compare, e.g., some works concerning class hierarchy, imbalanced class, fine-grained distinctions datasets, and methods. This could further refine the paper, providing deeper levels of analysis for some challenges and try to overcome them.
additional feedback: None
review_title: A Fine-Grained Dataset of Architectural Styles with Real-World Challenges

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: A dataset of consisting of ~10k images of Churches labeled with architectural style is presented. This is a challenging task requiring identification of detailed visual features and understanding the relation between them. At the same time the dataset presents challenges to existing recognition systems due to the small sample sizes, data imbalance, and viewpoint variability. 

The authors source the images from Wikipedia and filter images such that they depict the main building. The architectural style labels were constructed in consultation with architects and art historians, and thus automatic techniques to detect these styles in real images might enable large-scale studies in these domains. The authors present a detailed analysis of various design decisions that went the structure of the label space, dataset collection and post-processing. Several versions of the dataset are presented that vary depending on the number of class labels. Additionally, the authors identify a set of distinctive structures for which bounding-box annotations are provided. 

The authors also present several baseline experiments using a ResNet50 fine-tuned on this dataset. The classification accuracy is rather low for this problem, and the authors find that the increasing image resolution and class-balancing helps. 


strengths: The authors make well-motivated decisions for the design of the dataset which are described in detail in the paper. The dataset covers different types of images and labels compared to prior datasets. Hence is likely be of value to the community interested in designing computer vision techniques for analyzing architectural styles. 

The discussion of label noise and biases in the dataset will allow users of the dataset to scope the utility of the models trained on this dataset (e.g., geographical bias). 

The baselines are sensible and the low accuracy on this task might motivate novel computer vision and machine learning algorithms.
weaknesses: The utility of bounding boxes was not analyzed. For example, could the classifiers be improved by looking at regions within these boxes? Are existing detectors able to localize these parts from the limited amount of training data available? 
correctness: The claims are correct. The dataset construction appears to be sound and well motivated. The baseline experiments are based on the standard practice of fine-tuning ImageNet pre-trained classifiers. 
clarity: Yes the presentation of the paper is clear. The design of the dataset and baseline experiments are well motivated.
relation to prior work: The contribution is primarily a dataset. The authors discussion prior datasets and work on architectural style classification from the computer vision / machine learning community. 
additional feedback: None.
review_title: A dataset of architectural styles of churches;

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: In this paper, the authors introduce a dataset for architectural style classification sourced from Wikipedia. They additionally provide 631 bounding box annotations for 139 selected churches from four major categories. The dataset tackles challenges with small sample size, highly imbalanced class distribution, fine-grained distinctions, high variance of viewpoints, label noise, hierarchical structure of labels. The paper formalizes the task of architectural classification and its corresponding difficulties. 

strengths: The paper is well written. The dataset provides a large, comprehensive collection of images that is validated by a known expert on church architecture in Europe. The architectural styles are well curated and some images contain relevant bounding box information that could be used for tasks that require additional characteristic object annotation. 

The model exploration, while not novel, is extensive in its exploration of class balancing issues, resolution adjustment and explanation for the difficulty of classifying the images.

weaknesses: The dataset includes a hierarchical organization of labels, where only some images are labeled at the most precise level. The majority of 5678 churches are labeled only at the first level of the taxonomy, while only 81 churches are labelled at the most precise third level. The labels are hugely imprecise, and would limit many works tackling hierarchical classification. 

In addition, the authors acknowledged that the task of architectural style recognition is actually a multi-label problem. Casting the task to just the dominant style adds an additional layer of uncertainty in the classification task. It’s not clear whether visual features are indicative of class label, or just the time period. Without further historical context, could humans learn to classify this? 

The bounding box labels are unique, but not all objects of interest are labelled in a single image. This seems to be in contrast to MonuMAI [29], where every object is labelled, and has more flexibility for different vision tasks.

The authors mention that Renaissance is a notoriously difficult category to recognize since it has few images and overlapping architectures. Given the huge accuracy disparity for this category versus others, there is potential for this category to be augmented. 

Regarding the label noise, was the random selected images subset a uniform sampling or sampled per category? How were the number of images determined, since there might be a larger sample size for a certain desired confidence interval? If an image was labeled as uninformative, why was this label provided and could another expert correctly annotate it? How well could the baseline model classify these incorrectly labeled or uninformative images?

correctness: Yes. 
clarity: Paper is well written and easy to follow. 

relation to prior work: Prior work is discussed. Authors should additionally clarify how each of the canonical subsets they propose compare to prior work, with respect to attributes such as image count and architectural style count. 

additional feedback: See above.
review_title: Interesting multi-use dataset

Meta-review: The paper introduces a new dataset of ~9.5k images of churches annotated with fine-grained and hierarchical labels of architectural style. Reviewers felt that the dataset was novel and well-described, and would be an interesting testbed for fine-grained recognition. The primary weakness mentioned by reviewers was that the paper does not provide sufficient baseline classification results on the dataset. The authors provided additional results in their response to Reviewer roSv. In addition, the paper does not justify the utility of the bounding box annotations. On the whole, the AC feels that the benefits of this dataset for fine-grained classification outweigh the weaknesses. Congratulations on having your paper accepted to the NeurIPS 2021 Datasets & Benchmarks Track! The authors are encouraged to take the feedback from reviewers into account when preparing the final version of the paper.
Decision: Accept