Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes to use a differentiable approximation of the Heaviside step function (which determines where the prediction should be positive according to the input and threshold) to build a loss function that theoretically approximates Accuracy or F-score.  The motivation is to bridge the gap between training loss and evaluation metrics in binary classification.

In terms of the method, it has two parts. The first is to approximate the Heaviside function with a sigmoid or piecewise linear function with hyperparameter tau. Then this paper introduces the soft set membership to measure the degree of belonging to a certain confusion matrix set. In terms of theoretical support, the author proves that the new approximation of the Heaviside function is Lipschitz continuous and the loss constructed by their combination also makes the variation of each step in SGD optimization small. Then the paper proves that the loss that approximates the F1 metric converges to the F1 metric when the number of samples is infinite. Finally, in the experimental part, the paper experiments on tabular and image data and also explores the balance of precision and recall. The approximated loss of F1 score achieves higher performance than BCE loss and other losses.

strengths_and_weaknesses: 
Strengths:
1. The paper is novel and compares some related methods in related work and experiments.
2. This is a completed work, with both theoretical support and experimental results, and the authors honestly point out various deficiencies.
3. The presentation of this article is very good; the overall structure and specific details are relatively clear.
4. This paper provides a training loss that approximates the evaluation metric. The difference between training and evaluation is a problem of great interest.

Weakness：

1. Although this paper has compared adversarial-based methods in experiments, comparisons with other differentiable surrogate losses are lacking.

2. This paper can compare previous methods and the method in this paper for the difference in approximation error.  Previous methods include BCE loss and the previous methods to solve the difference between training loss and test indicators.

3. What kind of theoretical problems will the gap between the two-class training loss and test indicators bring about; this paper lacks the analysis in this regard.

4. There are some differences between the test sample and the training distribution. The closeness of the loss at training time to the metric at test time does not seem to solve this problem, and this article does not consider generalization error.

questions: 1. I suggest that this paper can give a figure that pairs and compare the original version and soft version and highlight the difference. Formula 2 and Formula 6 look identical (the difference is the small s).

2. Line 124, “By [11]”, I recommend not making numbers the subject of a quote.

3. What are the challenges of extending this research to multiclass classification?

4. I suggest that this article can also experiment on more kinds of data, such as binary classification on graph data.

limitations: The paper discusses limitations scattered throughout the paper. Besides the limitation summarized in checklist 1 (d), they also acknowledged other limitations.

In Line 119, the paper acknowledges that the mini-batch stochastic gradient descent (SGD) optimization method does not provide an unbiased estimator on non-decomposable metrics.

In Line 202, the paper acknowledges that the F1-Score computed from soft sets is a biased estimator for the expected true F1-Score for finite n.

In Line 249-254, they find that the loss that approximates AUROC does not work.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposes a method to directly optimize evaluating metrics, such as F1 and AUROC, at training time. The method includes a piecewise linear approximation of the 0-1 loss (heaviside function), and soft versions of F1 and AUROC metrics based on the diffierentiable approximation. Experiments show that the proposes method can achieve superior F1 and AUROC compared to models trained with binary cross entropy loss. 
strengths_and_weaknesses: Strength
- Binary classification is an important and frequent problem. 
- The proposed method can have wide application potential.
- The proposed method is simple and effective.

Weaknesses
- The novelty might be somewhat thin: approximating non-diffierentiable functions with differentiable ones are common. The theoretical grounding section might not be too superising (continuity, consistency). Proposing to directly optimize evaluating metrics at training time might be novel, but I think it is also studied, as those mentioned in related works and are compared against.
questions: 1. Based on my understanding, I think the technical part of proposed method includes a piecewise linear approximation of the heaviside function, and approximations of confusion-matrix metrics based on soft sets. I'd like to see an ablation study where these two metrics are evaluated separately. For example, if the heaviside function is approximated with sigmoid function, does the proposed "accuracy" metric coincides with BCE? 

2. Sec. 4.2 presents a consistency result. Isn't that also true for the BCE loss, at least for accuracy? I think to firmly demonstrate the advantage of the proposed method, consistency alone might not be enough, and one might want to consider the statistical (sample) efficiency?
limitations: Limitations an societal impact are sufficiently addressed.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: In binary NN classifiers, we train the models on BCE objective but at the test time, we evaluate the model's accuracy or AUROC instead. The authors propose a differentiable approximation to the test time metrics (like Accuracy/AUROC/F1 score) so that we can use them for training the model as well. The authors evaluate the new loss function on 5 datasets with varying levels of class imbalance and show that the models trained with proposed loss have better generalization.  
strengths_and_weaknesses: Strengths:
- The introduction of a new optimization objective.
- The investigations seem thorough.

Weaknesses:
- Did not justify enough why this new optimization objective is useful. I understand this improves the metrics we are interested in measuring but are there any other benefits.
- The paper is hard to follow in some places.
questions: The following are my major concerns, it would be great to discuss this 
-	How expensive is the method? In terms of compute and time efficiency.
-	How robust is the model to slight OOD perturbations? 
-	Is this method scalable to large datasets? 

limitations: n/a
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This work focuses on the loss function of the binary classification problem and proposes the use of the differential Heaviside Function Approximation $\mathcal{H}$ and soft set membership as a loss to directly optimize desired confusion matrix-based metrics such as accuracy, F1-score, etc., instead of the commonly used surrogate objective BCE. The study focuses on the analysis of a sigmoid function and a five-point linearly interpolated function as an approximation, with some assumptions providing theoretical underpinning. Experiments are conducted on several unbalanced level datasets, and the results are consistently good or comparable in some scenarios.
strengths_and_weaknesses: Strengths: The strategy suggested in this article provides a number of advantages over earlier attempts to directly improve measures. It alleviates the batch size limitation and being faster for the adversarial approach [11] (cubic runtime complexity), and it supports a wider variety of metrics as opposed to WMW [33]. 

Specifically, the method can optimize metrics such as $F_1$, $F_\beta$, Accuracy, AUROC, etc. directly and balance the tradeoff of Precision and Recall, without increasing the training time complexity (linear). 

The authors provide a theoretical foundation for this method and empirically validate it using a variety of datasets with varying levels of imbalance. The linear approximation function was introduced in [27] as an activation function, but this work seems the first to use it in the training objective.


Weaknesses: The experimental results show that only some of the metrics have a significant improvement, while others do not. For example, AUROC has no unbiased estimator which leads to bad performance. This approach seems less directly applicable to multiclassification tasks.

questions: Results are observed when the evaluation measure is F1 and the loss is also F1*, however practically all studies on accuracy are comparable to BCE (tabular datasets) or the same improvement is reached using F1* and Accuracy* (image datasets). Also, it appears that F1* performs better on numerous evaluation metrics; does this contradict what the theory predicts?
limitations: Did not conduct research with human subjects
review_title: null

Meta-review: metareview: When training binary classifiers, one usually minimizes the (surrogate) binary cross-entropy loss (BCE), but evaluates on metrics such as F1, AUROC, or other confusion metric-based scores. The authors propose to instead combine a differentiable approximation of the Heaviside function with a probabilistic view of the typical confusion matrix values using soft sets to directly optimize F1 and AUROC at training time. The authors show that under certain assumptions the metrics computed over the soft-set confusion matrix values are asymptotically similar to the underlying true metric. Finally, the authors evaluate the proposed approximations on several unbalanced datasets and show competitive performance with respect to optimizing BCE.

While the reviewers outlined some weaknesses, they agreed that this work is relevant to the larger research community and presents a novel approach with potential to be used in practice. During the rebuttal phase the authors addressed the main remaining issues and I will recommend acceptance. Please incorporate all the information presented during the rebuttal phase into the manuscript. 
confidence: Certain
Decision: Accept