Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper tests methods for training a classifier for predicting trustworthiness-- formulated here as predicting whether a given image will be classified correctly or not by an target classifier -- on large scale datasets and finds that using existing loss functions like uncertainty estimates/focal loss on this binary classification task cannot detect untrustworthy samples, all samples are classified as positive resulting in 100% FPR (0% TNR). They attempt to fix this shortcoming by introducing a steep slope loss which essentially separates features w.r.t. correct and incorrect predictions from each other, and the loss is bounded in its magnitude. The paper extensively tests whether its possible to predict trustworthiness with two oracle/target classifier combinations on Imagenet and its stylized/adversarial test sets.
main_review: Strengths:

2.1) Clear and well-presented: The contributions are clearly stated and well-motivated, acknowledgement is given to the existing body of work to the best of my knowledge. Experiments are performed on large-scale and existing loss functions fail to differentiate between samples. The paper then proceeds to describe a sufficient solution, results are clear and well-presented-- it seems a good step forward.

2.2) Very significant results and interesting transferability: The increase in TNR seems very significant compared to other loss functions, essentially going from near 0 to very significant distinguishability--which says steep slope loss is very useful. The results hold true for stylized and adversarial sets as well with equally significant degrees. The phenomena of ViT being able to detect RSN trustworthy samples far better than the other way around is a very interesting observation, the difference seems stark enough to go beyond simply it having more classification power. 

2.3) Reproducible & nice ablations: The code is provided alongside with detailed readme which I wanted to highlight  (I didn't verify the code but it should work well). The ablation with the balanced loss was well-though of and addressed the question I was thinking of.

Weaknesses:

3.1) Formulation unconvincing [Critical]: In my opinion, trustworthiness of a classifier’s output depends on the classifier. Consequently, the input to the trustworthiness predictor should have that output of the target classifier passed (softmax probability/logit etc) alongside the image. This formulation has only the implicit encoding by using supervised samples of a classifier without providing explicit knowledge about the classifier itself seems badly designed (in principle) and I strongly oppose this.

3.2) Unfair comparisons [Critical]: Past works to the best of my knowledge the trustworthiness (typically used for out-of-distribution detection rather than rejecting test samples) is determined by the probability outputs of the target classifiers (a function of the target classifier). The calibrated probability estimates for the same are obtained by using loss functions like cross-entropy (with some calibration) {13} and focal loss [1] -- not training a separate binary classifier with these loss functions as done here, along with assumption of having access to just the image. Recent (parallel) work [2] has benchmarked these methods on large-scale datasets which indicates these scale well to larger datasets. A fair comparison in my opinion should be with (older-- {13},[1]) classifiers using the output of target networks to determine trustworthiness. 

Overall:

The paper is well-written and the improvements are strong. The proposed approach seems simple, accurate and interesting. I mainly have the above concerns regarding the framework introduced and the comparisons made. Overall, I am on the fence but will be willing to increase my rating if the weaknesses are addressed. Note that I might have missed important contribution novelty (summary/strengths) or misunderstood claims (weaknesses) since I am not familiar with the uncertainty/OOD/classification-with-rejection domain.

[1] Jishnu Mukhoti, Viveka Kulharia, Amartya Sanyal, Stuart Golodetz, Philip H. S. Torr, Puneet K. Dokania, Calibrating Deep Neural Networks using Focal Loss, NeurIPS 2020

[2] Rui Huang, Yixuan Li, MOS: Towards Scaling Out-of-distribution Detection for Large Semantic Space, CVPR 2021

Post-rebuttal:

I am not convinced by the rebuttal and choose to be skeptical of the proposed approach. 

I think the crux of disagreement lies here:

 -  The trustworthiness scores are indeed dependent on the classifier.

The rebuttal and the paper explicitly say it only depends on the image input which makes it a very different problem formulation departing from previous literature [r1]. I want to stress that the difference is not between access to intermediate layers or the last of the unknown classifier, but that there should be access to the unknown classifier at all. [r1] in contrast does assume access, infact stating that they fix the unknown classifier and make it accessible to the trustworthiness predictor. 

In this work as I understand, ideally they should have trained another classifier to predict trustworthiness (of the unknown classifier) given only the soft information of predictions by that unknown classifier on training data. Otherwise it's confusing how a ViT unknown model, ResNet trustworthiness predictor could be used. I hope this clarifies the point I was trying to make.

(Note that [r1] chooses to take the intermediate layers but could just additionally have incorporated the last layer as well, however in this work they cannot assume access to the underlying classifier.)

The defense of miscalibrated classifiers and adversarial attacks seem not convincing to me for two reasons:

- Calibration has been surprisingly accurate and existing methods seem to alleviate the problem to a large degree on both in-distribution and OOD data. It seems like a more straightforward comparison to make if allowed access to the classifier-- which the authors could just say is unfair because that formulation specifically requires access which they cannot assume (which is my original objection).
- Adversarial attacks can be made on the trustworthiness predictor as well. I am pondering why dependence on the classifier makes it different, the trustworthiness predictor could ignore the classifiers outputs entirely and choose to rely more on the image.
limitations_and_societal_impact: No substantial negative societal impact that must've been discussed in my view but the section was missing from the draft.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: The paper proposes Steep Slope loss, a loss function that is designed to improve performance in the binary classification task of prediction trustworthiness. The paper reports that common loss functions used in this domain such as cross-entropy loss, focal loss, and true classification probability loss (TCP), all suffer from an issue where correct predictions tend to dominate over incorrect predictions, due to issues of class imbalance and evaluations on simple datasets where classifiers have achieved very high accuracy. 

Steep Slope loss solves this issue by controlling the slope parameters on both the negative and positive classification thresholds with two hyperparameters that control the derivative and gradient updates for datapoints that are correctly classified and incorrectly classified separately. These two controlled slopes are a function of the signed distance of the output of the classifier from the hyperplane of classification.

The paper evaluates this loss on the MNIST, CIFAR, and Imagenet datasets and shows empirical improvements on multiple binary classification metrics such as AUPRC, AUROC, and FPR95, and importantly, SS loss shows marked improvements in the TNR, therefore being more robust to detecting incorrectly classified points.
main_review: ### Strengths of the Work - 

The paper is well written, motivates the problem of trustworthiness classification well, and derives a well-argued formulation for Steep Slope loss. The empirical results are quite strong, and the datasets studied are comprehensive as well.

The formulation for SS loss seems like it will be quite beneficial for many binary classification tasks as well, especially in order to improve model calibration on in-domain data. This is a promising loss function to add to the family for more than trustworthiness applications as well. 

The comparisons to class-imbalance loss and clear reasons for why SS loss is a better choice also adds to the strength of the argument in a domain where the "positive class" and "negative class" statistics are unknown and depend on the accuracy of the classifier being studied. 

Showing the robustness of this loss to unseen domains such as the stylized and adversarial Imagenet test sets also strengthens the method, and the TNR numbers in Table 2 are quite impressive compared to the baselines.

### Comments and Suggestions - 

The paper is a good contender for acceptance in its current form. The appendices are comprehensive, and the following points, if addressed, would strengthen my opinion of acceptance further.

1. It would be interesting to see a numerical analysis of gradients of the Steep Slope loss function, whether there are vanishing gradient issues, or numerical stability issues due to the exponent.
2. Another potential test dataset to evaluate the robustness of SS loss could be the Imagenet-Corrupted datasets, defined in [1]. Trustworthiness as a function of dataset shift can be an interesting property to map, to strengthen the adoption of SS loss to domains where there can be covariate shift from the training regime to an unknown test regime.
3. Why are the TNR results so sensitive to the normalization by $||w||$? I would be curious to know if the authors have a hypothesis for why it is better to normalize the affine outputs with a norm of the weights?
4. The paper mentions that all the oracles were trained with the same hyperparameters (LR, weight decay, momentum etc). This doesn't make much sense to me, as each of these loss functions have very different training behaviours and gradient updates, and the number of steps required for convergence would be different. Further, since the loss  landscape for these loss functions are different, the optimal hyperparameter space would be different for each of the loss functions. It might be that the SS loss is in a good hparam space for the current hparams, but the other losses are not.
5. Could the authors clarify how the picked the values of $\alpha^+$ and $\alpha^-$ for different datasets, or point me to the relevant section in the paper in case I missed it?
6. From the histogram plots, it seems like that a lot of True Positive Examples get misclassified, with the tradeoff that more True Negative Examples are accurately classified. It would be good for the authors to discuss the pros and cons of this tradeoff, especially in real-world applications of trustworthiness (medical applications etc), which can strengthen arguments for their method.
7. A further comparison to non-loss functions method in the trustworthiness sphere could be good, such as the trust score [2].

Nitpicks - 

Line 45: "state-of-the-art"

Line 84: should be $f_\theta'$ instead of $f_\theta$

Line 136: "hyperplane"

Line 137: $x_{\text{out}}$ instead of $x_{\text{in}}$


### Originality of the work

To my knowledge, the formulation of the SS loss is novel, and the parametrization of positive and negative slopes to aid with the class imbalance present in trustworthiness classification has not been done before. There is rich literature in the field of proper scoring rules and binary classification for parametrized binary loss functions [1], but I have not seen any research in the field of trustworthiness that makes use of these parametrized loss functions. This work has definite novelty in scaling the methods to Imagenet and ViT, whereas previous work in this field focused on MNIST and CIFAR, which are much easier tasks with higher classification accuracy.

### Quality and clarity

The paper is well-written, hyperparameter tuning and ablation experiments are comprehensive, and the proof for the generalization bound seems correct to me. The authors have picked valid metrics to display the pros and cons of their method, though a further discussion on balancing the tradeoff between TPR and TNR would be good.

### Significance

In my opinion, the empirical results are impressive and push the envelope for binary classification in the trustworthiness sphere. The implementation of the loss function is also straightforward in a standard deep learning package, and can be plugged into existing pipelines with a one-line change. The authors also perform experiments on a large scale dataset such as Imagenet and report state-of-the-art numbers for their method and some baselines. The risk analysis plots are also an additional way of testing the robustness of these methods, and further results on synthetic and adversarial versions of the test dataset should be considered the norm for future analysis in this space.


limitations_and_societal_impact: The authors have responded N/A to this prompt in the checklist. However, I would advise the authors to add a small discussion about the importance of trading off between TPR and TNR for critical applications such as in medicine, where the robustness of their method to TNR would make a strong statement.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors find that past trustworthiness predicting methods are prone to overfitting to the training set where correct predictions are dominant. The label imbalance makes the trustworthiness predictors over-confident to even incorrect predictions. To solve the problem, this work proposes a new training loss named steep slope loss, which aims at improving the generalizability of trustworthiness predictors. The intuition behind this is to emphasize the negative class during training. The experiments are conducted in ImageNet. The results show the shortcomings of past methods with cross-entropy loss, focal loss, and TCP loss and then demonstrate the effectiveness of the proposed method. 
main_review: *Strengths
- The paper reveals the shortcomings of past works in large-scale datasets like ImageNet, which is an interesting problem.
- The paper proposes an effective method to improve the generalizability for trustworthiness predictors.

*Weaknesses
- The paper lacks a theoretical study about why the proposed loss is better than the previous methods.
- It is not clear to the reviewer the empirical and theoretical difference between the proposed method and simply up-weighting the negative samples in trustworthiness learning.
limitations_and_societal_impact: Yes, the authors have adequately addressed the limitations.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper proposes the steep slope loss for learning trustworthiness under a supervised training setup. A pre-trained neural network is further trained on the steep slope loss to learn trustworthiness as a binary classification problem. The approach is evaluated on ImageNet for a ResNet and a ViT model. In addition, evaluations on MNIST and CIFAR-10 are provided for comparisons with the related work. In all experiments, the proposed approach shows good performance.

main_review: Paper strengths: 

+ The paper addresses an open problem where there is not a standard solution. It's well-written and mostly easy to follow.

+ The steep slope loss is well-motivated. The visual illustration of the loss in Fig. 2 helps to immediately follow how it works. 

+ The evaluation relies on the standard metrics for the presented problem. Also, recent approaches are included in the evaluations.  It's clear that the proposed loss works well for most of the metrics. 

+ The paper is the first one to perform a large-scale evaluation for trustworthiness estimation.

+ Unseen domain evaluation: the proposed approach makes use of trustworthiness estimation for identifying wrong predictions in a second domain. This is a novel application for trustworthiness that has not been presenter before.


Paper weaknesses: 

- The small-scale evaluation is important for the paper because one can compare it with the prior work. They should have been part of the main paper.

- It's not clear whether the trustworthiness estimator (oracle) and the classifier are two separate models. If so, it's important to discuss the inference time of the presented algorithms. The trustworthiness estimator would probably double the inference. 

- Comparison with Trust Scores (To trust or not to trust a classifier, NIPS, 2018): The trust scores is a related approach for trustworthiness estimation. It would be supportive for the paper to compare with this approach.


Improvements:

- line 96: m is not defined.

- Eq. 4 / Eq. 5: what is the first minus?


Overall: 

The paper presents an interesting approach for trustworthiness estimation. The proposed loss function learns when a deep neural network makes untrustworthy predictions. The main innovation of the paper is the evaluation of ImageNet using two complex models. The proposed study brings trustworthiness estimation closer to real-world problems. It's the first one to consider ImageNet with recent network architectures such as the transformer.


After rebuttal comments:

The rebuttal addressed my points. In addition, the major concerns on the theoretical study, the difference between the proposed method and negative sample up-weighting, and Imagenet-Corrupted evaluation are discussed in the rebuttal. The proposed approach is the standard evaluation protocol for trustworthiness estimation. The approach does not have to be evaluated for out-of-distribution detection. The current protocol is completely fine and similar to the prior work. For these reasons, I retain my positive evaluation since the paper has enough merit to be accepted. 

limitations_and_societal_impact: -
review_title: null

Meta-review: This paper develops a novel loss function for predicting the "trustworthiness" of a classifier.  This involves training a separate classifier, on existing predictions and corresponding outcomes, that makes a binary prediction about whether a predicted class can be trusted.  The authors show that existing loss functions do not generalize well and instead propose a novel steep slope loss.  The authors present theoretical generalization bounds and empirical evidence on a variety of non-IID validation sets that their method outperforms existing losses.  Overall the reviewers thought that the paper was well written, the contributions significant and the empirical evidence convincing. 
 One criticism was regarding the amount of access required by the method to the classifier of interest (there was significant debate about this and seems to have been resolved to an acceptable level).   Two reviewers voted high confidence accepts (7, 8) and two low confidence borderline scores (5, 6).  Thus the recommendation is to accept the paper.   Please try to incorporate feedback from the reviewers (e.g. clarification about access to the classifier) in the camera ready version.
Decision: Accept (Poster)