Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round1

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This paper presents a dataset, named Pl@ntNet-300k, for plant image dataset, and it also recommend two set-valued evaluation metrics associated with the dataset.
strengths: The paper proposes a comprehensive dataset with high intrinsic ambiguity and long tailed distribution, and the paper is well written and easy to follow.
weaknesses: It is better to provide more experiments to validate the contributions and the proposed dataset. Other backbones should also be used.
correctness: The paper is easy to follow.
clarity: The paper is easy to follow.
relation to prior work: Yes, and it is better to provide more analysises.
additional feedback: It is better to provide more analysises of previous papers, and more experiments should be provided, such as different backbones.
review_title: Pl@ntNet-300K: a new plant image dataset for the evaluation of set-valued classifiers

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This paper presents a novel image dataset with high intrinsic ambiguity specifically built for evaluating and comparing set-valued classifiers.  The author recommend two set-valued evaluation metrics associated with the dataset (top-k and average-k) and provided the results of a baseline approach based on a resnet50 trained with the cross-entropy loss.
strengths: The proposed dataset has a strong class imbalance, meaning that a few species account for most of the images. And many species are visually similar, making identification difficult even for the expert eye. These two characteristics make the present dataset well suited for the evaluation of set-valued classification methods and algorithms. 
weaknesses: The author only used a resnet50 as baseline. I wonder it is not the SOTA model in set-valued classification. 
correctness: Yes. 
clarity: Yes, the paper is well written.
relation to prior work: Yes, but it would be better if there are more discussions and comparation between set-valued classification and Fined-Grained Visual Classification (FGVC).
additional feedback:  It would be better if there are more results about SOTA methods performance on the proposed dataset.
review_title: Good paper

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This paper presents a plant image dataset, called Pl@ntNet-300k, and provides detail baseline evaluation based on resnet50. Pl@ntNet-300k is much more comprehensive than previous datasets, which is designed to remain representative of real-world ambiguity ​across classes, combined with the long tailed distribution, makes it more challenging to set-valued classifiers.
strengths: - First effort at creating a comprehensive dataset with high intrinsic ambiguity and long tailed distribution. Will accelerate research. 

- Clear explanations and illustrations.

- In general the paper is easy to follow.




weaknesses: - Experiments are insufficient to validate the contributions.



correctness: This paper described the construction of the dataset, image selection and train/test/val set division is in reasonable way.

clarity: Writing quality is good.
relation to prior work: Yes, this paper compares the proposed dateset with previous contributions.
additional feedback: - It is better to show results based on other classifiers along with resnet50, such DenseNet, MobileNet.
- Make sure the images in the dataset are not recompressed with a lossy-method.
review_title: While there are no new technical insights, this work represents a good initial effort to help the research community to evaluate classification methods in real-world ambiguity, I rate it as weakly accept. 

Meta-review: While the proposed dataset could be quite valuable to the community, there are a few experiments that would significantly strengthen the paper. Currently, it's not clear how difficult/easy the dataset is, what the relationship is between avg-k and top-k, and whether ImageNet models would perform well on the dataset without much modification.

Overall, we recommend the authors resubmit after adding the following experiments:

1. scatter plot of ImageNet versus PlanetNet accuracy for at least 20 models covering decent range of accuracies
2. scatter plot for top-k versus avg-k metric to see if one is just a function of the other
3. some estimate of Bayes risk (e.g. by evaluating a trained human labeler) so that we have a sense of how difficult /easy the dataset is
Decision: Reject