Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round2

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper proposes the MET dataset for instance-level recognition of artworks. The MET dataset poses a number of different challenges and has great significance in academic research.
strengths: + The MET dataset is the first large-scale dataset in the task of instance-level recognition of artworks.
+ The paper is written well. The collection and annotation of the dataset are described in detail.
+ The MET dataset poses a number of different challenges and has great significance in academic research.
weaknesses: - The MET dataset is collected from multiple sources. The authors are encouraged to evaluate the performance of domain adaptation.
-  It's confusing that the val/test set is very small both on image level and class level compared with the training set.
correctness: Yes
clarity: Yes
relation to prior work: Yes
additional feedback: See part "Weaknesses"
review_title: Instance-level Recognition for Artworks: The MET Dataset

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This work introduces a large-scale dataset for instance-level recognition for artworks. The authors provide benchmarks on the dataset.
strengths: 1. The proposed dataset is a large-scale dataset consisting of about 400k images from more than 224k classes. The proposed dataset could be useful for testing algorithms for representation learning.
2. The authors also provide benchmarks and baseline models on the proposed dataset.

weaknesses: It would be great to provide more details on the labeling process. For instance, how long will it take to label the images?
correctness: Looks correct.
clarity: Yes.
relation to prior work: Yes.
additional feedback: This work introduces a large-scale dataset for developing representation learning algorithms.
review_title: A dataset for artworks

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This well-written paper presents an ILR dataset on artworks and summarizes the comparison to the existing ILR dataset. Different kinds of methods are compared and evaluated properly in the paper.
strengths: 1. Various methods are compared in the benchmark. Results and training details are well provided.
2. The proposed dataset has a much larger number of images and classes than the existing artwork ILR dataset, making it reliable for further research works in the future.
3. This paper is well written with well-designed figures and tables.
weaknesses: The proposed dataset is very similar to existing ILR datasets, which may have difficulties in promoting new research works. GLDv2 has a similar number of instances but with a significantly bigger number of images. Although there are differences between the two datasets as described in the paper, how they will influence existing ILR methods is uncertain. More analysis on how specific features of the proposed dataset influence the existing ILR methods is required.
correctness: The submission is a dataset with the benchmark which is constructed in a sound way compared with existing works. The benchmark covers different deep learning methods for proper evaluation.
clarity: This paper is very well written.
relation to prior work: The difference between the proposed work and previous works is well discussed.
additional feedback: None
review_title: A well-constructed ILR dataset on artwork. More analysis on the difference between the proposed dataset and existing ILR dataset is required. 

final_score: rating: 5: Marginally below acceptance threshold
review_content: summary and contributions: This paper proposes a dataset for instance-level recognition in the domain of artworks. The dataset includes training data collected from MET organization under studio condictions, and query data composed of images from MET collected by users and distractors. The dataset is large-scale and precisely labeled with no noise. Comparisons to existing datasets are clearly illustrated, which provides a clear position of this dataset. Experiments are performed based on both kNN and cross-entropy classifiers and for kNN, state-of-the-art feature representation strategies are evaluated. 
strengths: 1. The paper is large-scale in the domain of artworks and seldom datasets can be precisely labeled without noise at this scale. 
2. Comparions to existing artwork datasets are well conducted, which provides clear position of the proposed dataset. 
weaknesses: 1. The dataset may not raise common interest to the potential audience of NeurIPS. Comparing to other instance-level classification tasks such as face verification, place recognition, the artwork recognition receives obviously less attention.
2.  The authors claim the dataset is long-tailed. But in evaluation, techniques used to address long-tailed distribution are not incorporated, which makes the evaluation incomplete. 
3. The queries are collected from 39 visitors, which are relatively narrow and can make the evaluation biased. 
correctness: The dataset construction basically makes sense. My only concern is that the query images are relatively narrow as they are collected from only 39 visitors. In terms of experiments, more evaluations are expected in the flavour of long-tailed learning. 
clarity: The paper is presented in a clear and logical way. 
relation to prior work: Sufficient and clear discussion to prior work are provided. 
additional feedback: In terms of how to improve the paper, please refer to the above sections. More proper venues for this paper inclue ICMR, ACM Multimedia, and other vision related venues etc.
review_title: An Instance-level Recognition dataset that may not raise common interest from the audience of NeurIPS 

Meta-review: This paper presents the MET dataset, a cleanly-labeled large-scale image dataset in the artwork domain for the study of instance-level recognition in the domain of artworks. While reviewers agree that the dataset is well-constructed and the paper is clearly written, some raised the concerns that the scope of the dataset is somehow overlapping with GLDv2, and that the dataset seems to be of limited interest to the NeurIPS audiences because of it domain-specific nature. After the authors' response, one reviewer agreed to raise the score from 5 to 6 (borderline accept), although it is not reflected in the system. Then now the all four reviewers are in the positive side, two of them with clear accept.
I, the AC, agree that this paper will fit better to vision or multimedia conferences. Nonetheless, since there is no severe weakness in the paper, I think it is ready to be presented at this track.
Decision: Accept