Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes a number of desiderata for classification measures and examines common measures, such as accuracy, Matthew’s correlation coefficient, and F1 score, to show which metrics have desirable properties. Some properties include symmetry, distance interpretation, monotonicity, and a constant baseline. 
main_review: Overall, the results are numerous, if not all that difficult (from my admittedly quick glance at the appendices) and the discussion is fairly thoughtful. I’m not overly familiar with research specifically on classification metrics, so it’s unclear to me how novel some of these ideas are. In the case that they are at least somewhat novel, this paper would be a nice starting point for discussions on metrics, which I feel is somewhat lacking in the community.


limitations_and_societal_impact: I have two complaints and a societal impact comment.

My primary complaint is that I would have benefitted from additional structure in the main paper. Specifically, a distinct proposition here or there would have been nice, but the exposition is otherwise reasonable to follow.

A secondary complaint is that I would have liked to have seen some more examples and discussion balance and imbalance in the data. There is a slight discussion in the ImageNet discussion, but I think there could be room for a bit more, given that class imbalance is one of the motivating factors behind the use of some metrics (e.g., the ever-annoying F1 measure).

This could likely be a paper on its own, but I think the paper could potentially have negative social impacts. The reason is essentially fairness: how is selecting a desirable classification measure in practice going to affect different groups? I certainly don’t view this as disqualifying--I just bring it up because we’re asked to think a bit more about these issues now. I just think a mention should be made that in practice one may also need to consider other properties of the measure.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The core question studied in this paper is how to choose a best performance measure (termed as "index" by the authors) in all situations for binary and multiclass classifications. To answer this question, the authors define several properties that are important for considerations when one wants to choose a good index. Then many common indices (performance measures) in classification tasks are analyzed to see if they satisfy each of the properties defined, including Accuracy, F-measure, and others. A impossibility theorem is proved to show that some desirable properties cannot be satisfied at the same time for a given index. With this theorem, a family of indices is proposed that satisfies all good properties except one.
main_review: Overall I think this is an interesting and comprehensive study. The guidelines (properties) provided in the paper would potentially benefit practitioners who need to choose a performance measure for a given task. Out of the properties considered, the *constant baseline* is very important and useful. However, my main concern is about the novelty and significance in the machine learning community. See below for the detailed comments.

Originality.
There are two main issues.

1) Overall I feel that this paper is an incremental work and its novelty is of concern. This paper is based on a recent paper [1] (as the authors have acknowledged) that provides a systematic analysis of different indices for the clustering task. The authors transfer most of the properties defined in [1] to the binary and multiclass classification tasks. The transfer to binary tasks is trivial (as the authors have mentioned). One of the new contribution, that the authors claim, is to extend the properties and analyses to the multiclass case. However, I fail to appreciate how multiclass case is different from the clustering case (besides the setting of course), what the hurdles are when one transfers the properties and analyses to multiclass tasks, and whether new techniques are used. It would be more clear if the authors could elaborate the relation with [1]. 

2) The second concern is about the position of this paper in the machine learning community. Although this paper might be the first to present a comprehensive systematic study of different indices and properties in binary and multiclass classifications, many prospects have been touched more or less in earlier works.  I have read the related work section in the supplementary material, but I cannot fully recognize the position of this work, partially because much cited work is not from the usual machine learning venues, and I do not have enough information to bridge this paper with the background. Therefore, I decide to raise this concern to see if other reviewers could help better position this paper. Also, the authors should provide more background and related work from the machine learning venues.

Quality.
This work is theoretical and overall is technically sound. The claims are backed by theory and experiments. 

Clarity.
This paper overall is clear, and is able to communicate the main message to the reader, though some improvements are still needed. Here are some suggestions.

1) Some terminologies need more explanations, for example, *reference* (true labels), *candidate* (predicted labels), *index* (performance measure), *validation index* (target performance measure). I do not know whether these terms are standard in other community, but it is the first time I have seen these terms. It would be helpful if the authors could use standard terminologies in machine learning or explain the terms so that this paper would be more accessible to people in the machine learning community. As I have said earlier that I think this work may be beneficial for practitioners working on machine learning, it is important to make this paper more accessible.

2) Tables need more details in the captions to help understand the content. Please consider moving some of the details from the body of the paper to the caption of a table.

Significance.
This paper is a comprehensive systematic study of different indices and properties in binary and multiclass classifications. The properties studied in this work can potentially serve as guidelines for practitioners who use machine learning to help them find a good performance measure for a given task. However, the use of some nonstandard terminologies and the lack of background and related work in machine learning can have negative effects on the significance and accessibility of this paper.

Take the above into consideration, especially my concern about the novelty, I would vote for a weak accept.

---

Other questions and comments:

1) Is index A with more properties ($P_A$) strictly better than index B with less properties ($P_B$ where $P_B \subsetneqq P_A$)? Should practitioners choose the index with as many properties as possible?

2) What might be the slide effect for an index with many properties? Is it harder to compute?

---

Typo:

Page 2, line 41, "MCC" -> "CC"

---
Citation:

[1] 
Gösgens, M.M., Tikhonov, A. &amp; Prokhorenkova, L.. (2021). Systematic Analysis of Cluster Similarity Indices: How to Validate Validation Measures. <i>Proceedings of the 38th International Conference on Machine Learning</i>, in <i>Proceedings of Machine Learning Research</i> 139:3799-3808 Available from http://proceedings.mlr.press/v139/gosgens21a.html.


limitations_and_societal_impact: Yes, but the authors are encouraged to include a disclaimer/discussion on the fairness of different indices.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposed a systematic approach to choosing a suitable validation index for classification tasks. The authors considered several binary and multiclass indices, including new ones. They examined the indices with five sets of properties based on the study [6,18_Supplementary]. For each index and each property, they formally proved or disproved that the property is satisfied. They also presented the impossibility theorem for classification. 
main_review: At this stage, I consider this paper "marginally above the acceptance threshold" due to its overall contributions. 

Originality: 
It does a good job at studying various validation index for classification tasks with the five sets of properties. I like the part of the derivation of the impossibility theorem for classification because it is a good knowledge for our design of index. In addition, their proposed indices present better satisfactory of the properties than the other existing indices. 

Quality: Good. However, my main question is about the selection of the properties. The five sets of properties were transferred from clustering in [6]. The authors are better to discuss the extra properties in classifications. For example, this paper misses the property of Robustness to Imbalance (IMB), shown in [18_Supplementary] for classification. In fact, IBM seems to be the critical property in terms of "class ratio changes (Brzezinski, 2020)" in class imbalance problems. If the paper includes this part and makes the the IBM for the new indices, I will give a much higher rating. For this reason, the word "adequately" in the Abstract is an overstatement.   

Brzezinski, D., et al. "On the dynamics of classification measures for imbalanced and streaming data." IEEE transactions on neural networks and learning systems 31.8 (2019): 2868-2878.

- p2. a_i=sum from {j=1} to {m} is not suitable for c_00. 

Clarity: Good. 

Significance: Yes in terms of the five sets of properties, but missing the IBM. 
limitations_and_societal_impact: The authors addressed the limitations in the final section.
review_title: null

Meta-review: Throughout the discussion the reviewers agreed that the paper provides an interesting and comprehensive study of performence measures, and that it is of interest to the NeurIPS community. We urge the authors to revise the paper according to the comments provided by the reviewers in the rebuttal.

Thanks for submitting your work to NeurIPS!
Decision: Accept (Poster)