Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round2

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The paper introduces a challenging image classification dataset with 300K+ plant images grouped into 1081 classes (species). This dataset has difficulties which is often observed in practical applications of Image classification. The dataset is so difficult that an expert human annotator gets only 80% times correct. The species are grouped in to Genus, so there is a hierarchical structure in the way classes are related. Number of instances per classes is skewed which brings a huge class-imbalance issue which makes the dataset more challenging. 
strengths: - A huge dataset on a domain which is not covered much in other image datasets.
- A good set of experiments with recent classification models were done to showcase the difficulties in the dataset.
- Experiments shows that state-of-art techniques has a lot more gaps to fill to reach human annotator accuracy. 
- Authors seems to have addressed all the concerns raised by the meta-reviewer in the first round (this is what I learned from the supplementary material.)

weaknesses: - I think the "Plant Village dataset"[1] should be cited in this paper. It is a plant image classification dataset with plant diseases and species as classes. If there is an overlap of species between these datasets, can they be combined to create better classifiers?
- Writing:
    - Set-valued classifiers seems to be a recent paper and it is not very clear why this particular dataset be the best case for that experimental setup. Having this experimental protocol in the title is taking the focus away from the dataset.
    - There are much more details on algorithm and loss functions provided, which personally I feel does not fit well in a dataset paper.


[1] Sladojevic, Srdjan, et al. "Deep neural networks based recognition of plant diseases by leaf image classification." Computational intelligence and neuroscience 2016 (2016).
correctness: seems correct.
clarity: Yes, the paper is well written.
relation to prior work: Missed an important reference on "Plant Village" dataset missing. Refer to "Weaknesses".
additional feedback: NA
review_title: Big Image classification dataset to identify species of a plant.

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: The paper presents a long-tailed image dataset with high intra-class variability which is big enough (306,146 images) for experimentation with ML techniques, and contains many types of uncertainty. The authors further proposed performance metrics to evaluate models trained on the dataset that accounts for its uncertainties. The author also provides many baselines and analyses into the relationship between proposed metrics.
strengths: -	The paper proposes a big enough dataset for experimentation with ML algorithms. The dataset could benefit the research community.
-	The authors analyzed the relationship of top-k accuracy and mean-k accuracy. It showed that their relationship is not trivial.
-	Relationships between the performance on the ImageNet dataset and the proposed dataset of baseline models are shown.
-	The discussions of Epistemic and Aleatoric uncertainties are well written.

weaknesses: -	The dataset contains species that have at least four images. It is split into train, valid, and test with a ratio of 8:1:1. It is unclear how the author deals with species with a small number of images (4-5 images). How many are in the train, valid and test set for such cases?
-	Section 4.2 Baseline lacks training details like training epoch and learning rate schedule.
-	The performances of baselines on average-k metrics were not shown in the paper. Figure 6 has mean average-5 accuracy as the y axis, but the caption is top-5 accuracy.
-	The reviewer wonders how many annotators participated in labeling the original dataset and their consensus. The paper would benefit from the discussion regarding inter-annotator consensus in section 3.3 Aleatoric uncertainty.

correctness: -	The proposed dataset is created from a bigger dataset such that the nature of the original dataset is kept, i.e., long-tailed distribution, intra-class variability, and ambiguous data points.
-	The proposed metrics: Top-k accuracy and Average-k accuracy, are suitable for evaluating set-valued classifiers.

clarity: Although some minor details are missing, the paper is well written and easy to follow. 
relation to prior work: The author compares the proposed dataset with other related datasets on Fined-Grained Visual Categorization in many attributes such as long-tailed, intra-class variability, and others.
additional feedback: According to the authors, which metrics are the best to measure the performance of a model on this dataset, top-k accuracy or average-k accuracy?
review_title: A good dataset for evaluating set-valued classifiers and long-tailed learning but lacking some minor details

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: Authors describe a dataset formed by 300K images of plants that are organized according to a taxonomy. The task associated to the dataset is image classification, with the main challenge that there is a high imbalance among many of the plant categories. Baselines are evaluated and compared to the performance of botanists. 
strengths: * Authors release a relatively novel  dataset for the classification of plants, where categories are associated to a taxonomy and where the associated classification problem is challenging. 


weaknesses:  * Authors motivate the dataset as one that will allow studying set-valued classification, I think this motivation is weak (see detailed comments below), other aspects of the benchmark could be highlighted like: the inherent hierarchical classification task, the fine grained problem (mentioned only briefly), etc. 

* Authors argue on the importance of set-valued metrics, but they do not provide any result of a baseline optimizing such metrics. 

* I liked this dataset, but I could not determine/assess the relevance of the gap it pretends to fill, to me it looks like just another dataset (associated to an interesting problem and with their own challenges, which are not well justified) that can be introduced in a publication that describes a solution to the associated problem
correctness: I think the claims are correct, perhaps my only critic here is that some simple concepts are presented in a complicated way and they do not help in justifying/motivating the dataset.
clarity: The paper is clear (apart from the comments mentioned elsewhere)
relation to prior work: This could improve, in the current version authors focus only on fine grained classification tasks. But related problems (not necessarily datasets) should be contrasted, for instance, multi label classification, set based classification, hierarchical classification etc. 
additional feedback: * Please state the differences of set-valued classification with related formulations, eg., multilabel classification, set-based classification, hierarchical, link based classification, etc. 

* can you please define the right term of Eq (2)?

* Can the authors further elaborate on why it is important to approach the problem formulated in Section 2?, this is, motivation for this problem, why it is worth approaching it this way and not using multilabel classification for example?

* The information on metrics (4.1) is to detailed even when the metrics are quite common and intuitive (eq 5 is widely used as the authors mention and eq 6 is also widely used in multi label classification)

* Authors try several baselines but none of them tries to optimize the proposed evaluation metrics directly. This is one of the motivations of the dataset, yet authors still use cross-entropy as loss function. It would be interesting to know the performance of models that either optimize such metrics directly or other approaches that are appropriate for approaching this problem, e.g., methods for hierarchical classification, multilabel classification etc.
review_title: Potentially useful resource

final_score: rating: 5: Marginally below acceptance threshold
review_content: summary and contributions: In this paper, the authors introduce a classification dataset built from the database of the Pl@ntNet platform. The dataset has strong class imbalance and are visually similar, which pose as challenges for set-valued classifiers. They recommend mean top-k accuracy and mean average-k accuracy as evaluation metrics and provide baseline results from deep neural networks.

strengths: The baseline models for evaluation are comprehensive and provide insight into the different degrees of success for each model. The comparison to an expert botanist shows room for improvement for existing models.

Evaluation metric and analysis is well-defined with a justification for why the metrics were chosen.

Comparison to prior work shows that the paper's motivation is well-founded and is distinguishable from past datasets. An appropriate dataset split and acknowledgement of the task's difficulty was qualitatively shown.

weaknesses: I am mainly concerned about the significance of contribution, as the main contribution of the dataset is to randomly sample 10% of genera from the Pl@ntNet database. The evaluation metrics recommended are canonical classification metrics. 

In addition, there doesn't seem to be data cleaning involved. Though the authors retain uncertainty in the dataset, the variability involved in users taking different angled pictures of plants, different zoom/crop of plants, plants taking on different appearances depending on seasons (flowering time), is too high. Additional subsets of the dataset where this is cleaned, separated into different versions and accounted for (for example, Pl@ntNet-cropped, etc), is recommended. 

The authors also acknowledge that if a person photographs only the flower of a specimen of one of the two similar species, then it will be impossible, even for an expert, to know which species the flower belongs to. Though this is a reasonable task for set-classifiers, it seems misleading then for the authors to report mean top-1 accuracy for an impossible task in Section 4.3. When even the error rate for a botanist is above 0.2 for an average of 4,1 species returned, the task overall seems unreasonable. 

What is the label noise of the dataset? How are Pl@ntNet image labels validated? What is the least amount of labels per species in the train set? 

correctness: Yes. 
clarity: Yes, though notation breakdown could be more clear. 

relation to prior work: Yes. 
additional feedback: See above.
review_title: Dataset for set-classifiers 

Meta-review: Although the dataset shows unique aspects, there are critiques related to lack of proper motivation, missing design decisions (noise, sampling decisions, partitions, labeling, sanity check, etc.) and analyses. However after discussion with the authors, several critiques have been clarified. Overall the paper achieves the minimum score to be accepted for publication at NeurIPS data track.
Decision: Accept