Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper studies accuracy estimation from unlabeled data, in the presence of distribution shifts. They show that "dispersity" that captures the marginal distribution of the predicted labels correlates well with the accuracy. The paper assumes that the true label distribution is uniform and proposes a measure of dispersity that measures how much predicted label distribution deviates from uniform. This measure builds off of previous works on nuclear norm. Intuitively, this won't work when there is label shift and the paper shows that nuclear norm based estimation does indeed fail under large label shifts but is able to handle moderate label shifts. 
strength_and_weaknesses: Strengths: 
(i) The paper studies a very important problem of unsupervised accuracy estimation under distribution shifts. 
(ii) The paper considers an interesting idea and is well motivated. The paper is overall well-written and easy to follow as well. 
(iii) The paper performs very systematic and thorough experiments on a large number of datasets. 

Weaknesses:
(i) The paper can do a better job at synthesizing the results of the experiments, especially because they could reveal interesting insights about deep networks or distribution shifts, beyond just providing a method that improves accuracy estimation. For e.g. why is ObjectNet always off the line---is this because of the label shift?
(ii) The paper is missing some comparisons to recent related work: Baek et al. 2022 (agreement-on-the-line) and projection norm (Yu et al. 2022). Baek et al. is a recent paper, but projection norm was older. 
clarity,_quality,_novelty_and_reproducibility: Clarity: the paper is quite clear and easy to follow. I would recommend moving some intuition for why nuclear norm captures dispersity to the main paper, even though this is from prior work. It seems crucial to appreciating and understanding the proposed method. 

Quality: the experiments are exhaustive and claims are substantiated. The paper also addresses limitations of the method appropriately, and hence I rate the paper as high quality. 

Originality: the paper takes an existing idea of dispersity (and how it can be approximated by nuclear norm of the prediction matrix) and applies it to the problem of unsupervised accuracy estimation. It is a novel application of an existing idea. 
summary_of_the_review: This paper presents an interesting and intuitive idea of using dispersity in model predictions for estimating accuracy on unlabeled data from a shifted distribution. The results are compelling and I think this would be an interesting contribution to the ICLR community and inspire follow-up work. The paper could do a better job with exploring more interesting takeaways and insights based on the strong correlation between dispersity and performance. The paper could also benefit from exploration of other metrics (not based on nuclear norm) that also try to incorporate the marginal of the predictions to determine OOD performance.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a new heuristic to estimate the accuracy of a given model for OOD test observations without labels. It uses the nuclear norm (i.e., the normalized sum of singular values) of the prediction matrix (i.e., a matrix of softmax outputs for the test data) of a given model. The measure is motivated by results from previous work, which show that the nuclear norm integrates the confidence and dispersity of the model predictions. While the confidence was shown to be predictive for the OOD performance by previous work, the paper demonstrates that the dispersity is also a predictive feature. Empirically, the paper shows a strong correlation between the computed nuclear norm and the OOD classification accuracy on multiple datasets with synthetic and realistic corruptions. The proposed heuristic shows a slightly but consistently stronger association with OOD accuracy compared to existing measures across multiple datasets, models, and types of shifts. However, the nuclear norm performs worse on test datasets with imbalanced classes and seems to overestimate the OOD accuracy on randomly synthesized datasets
strength_and_weaknesses: **Strengths**

The paper is quite clear and easy to read. The goal is stated clearly. The empirical results are in line with the claims from the abstract and introduction.

It's helpful that the paper provides an initial motivation by showing the relationship between dispersity and OOD accuracy.

The association between nuclear norm and OOD accuracy is very strong and consistent across datasets. The used visualizations show the trend very clearly.

The paper is transparent about the limitations of the proposed heuristic and points out directions for future work.


**Weaknesses**

The presented setting for unsupervised accuracy estimation seems to be a special case of the unsupervised domain adaptation setup with only one training domain. It is not clear why the problem is not presented from the more general perspective of domain adaptation. To better assess generalization, the evaluation could consider more evaluation metrics, not only accuracy.

The results suggest that the nuclear norm performs best on datasets with balanced classes. However, on imbalanced test datasets, the predictive value of the nuclear norm starkly decreases, whereas at least one of the competing methods seems to be much more robust. In light of these results, it would be helpful to have more thorough ablations with respect to the performance on imbalanced datasets. For instance: What if both the training and test data were highly imbalanced? How well do the other heuristics (AC, ANE, DoC, MI) perform under imbalance and/or label shift? Can you provide an experiment that demonstrates that the nuclear norm can be adapted to deal with label shift?

The nuclear norm seems to consistently overestimate the OOD accuracy for randomly synthesized datasets (CIFAR-10-$\overline{\text{C}}$-Rand). Is there a particular reason for this? It would be helpful if you would also provide the performance metrics for each subplot in Figure 5 and Figure 7.

From the abstract, it sounds like the nuclear norm and its connection to confidence and dispersity would be a core contribution of the paper. Only at the end of the introduction was it indicated, and in Section 3.3 explained, that the nuclear norm has been used for similar applications (domain adaptation) and that the connection to confidence and dispersity was already established by previous work. I think it would be better to state the contribution more clearly/conservatively in the abstract.


**Questions and comments**

In Table 1, the performance improvements compared to existing methods seem relatively small in many cases. Is there a way to compute standard deviations or other uncertainty estimates to quantify the significance of these results?

At the beginning of Section 3.2, the prediction matrix is defined on the real numbers. Shouldn't it be defined on the interval (0, 1) instead?

In Section 3.2 it is suggested that classifiers that do not generalize well tend to give degenerate predictions by assigning test samples to some specific categories. Are there any insights into *which* categories typically correspond to degenerate predictions? Or is it quasi-random and depends more on the seed than on the actual categories?






clarity,_quality,_novelty_and_reproducibility: The novelty is somewhat limited because the nuclear norm has been used in the context of domain adaptation and the connection to confidence and dispersity was already established. To me, this was not sufficiently clear from the beginning; it was indicated at the end of the introduction and only became sufficiently clear in Section 3.3.

Both the clarity and quality of the paper are good. The analysis is mainly empirical but the experiments are comprehensive. However, I think the paper should provide more results for the case of imbalanced datasets and/or label shifts because this is a common issue in practice.

Reproducibility should not be a problem, as most of the experiments are based on existing models and datasets. However, it could be improved by providing code for all experiments.
summary_of_the_review: For the goal of unsupervised accuracy estimation, the paper proposes a simple heuristic that shows a strong association with OOD accuracy across many models, datasets, and types of shifts. It is a solid paper with a thorough empirical evaluation, but the utility for imbalanced datasets is not sufficiently clear. It could be stated more clearly that the idea is heavily based on existing work, and the connection to the more general setup of domain adaptation could be clearer.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The authors of the paper argue that confidence and dispersity are two important factors regarding the performance on the out-of-distribution datasets. Along this line, the authors propose to use the (normalized) nuclear norm of the prediction matrix as a metric for the generalization capabilities of the models. Various experiments on image datasets show there exists a strong correlation between the accuracy on the out-of-distribution data and the proposed metric.

strength_and_weaknesses: Strength: it is an interesting thought to consider the distribution of the predicted labels and use them as a metric to evaluate the generalization capability for the out-of-distribution data.

Weakness: I do have some serious concerns about the underlying assumptions used in the paper. To me the dispersity assumption may be too strong for a lot of real applications. For those image data sets used in the paper (imgnet, cifar10, CUB-200), those assumptions seem perfect since the class labels on the shifted test data are distributed relatively evenly. However for real world applications, the labels may be distributed very differently. For example some classes are rarely observed, and some classes are observed very frequently. The power law and long tail effects are prevalent. In such a situation, the dispersity assumption used in the paper is no longer valid, and thus the proposed metric may not work. This makes me wonder if the good performance shown in the paper is only an artifact of the “label-balanced” data sets.

clarity,_quality,_novelty_and_reproducibility: Some typos:
1. The second line in the Notation of Section 3.1: \Delta_k was introduced in the where clause without being used before in the sentence
2. The fourth line in the Notation of Section 3.1: it should be \Delta_k instead of \Delta_n

summary_of_the_review: Overall my review for this paper tends to be negative due to the major concerns I stated in the previous comments. If the good empirical performance shown is mostly due to the artifacts of the relatively label-balanced datasets then I feel maybe it is better to introduce more new datasets that have highly unbalanced classes to test the robustness of the models instead of making use of the artifact for a little improved numbers.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: In this paper, the authors propose a new way to try to predict a model's classification accuracy on out-of-distribution test data without access to the labels.  Traditionally, one might consider model confidence or the entropy of the predictive distribution.  The authors argue that a notion of dispersity would be more effective, and propose measuring dispersity by computing the nuclear norm of the example x class prediction matrix.  The authors show across a variety of datasets and corresponding out-of-distribution test sets that the nuclear norm is more strongly correlated with classification accuracy than other competing metrics.  

The overall review scores were quite mixed with 8, 6, 3 and averaged just below borderline.  Overall, the reviewers felt that the paper was well motivated, well written and easy to follow.  Two reviewers found that the experiments were thorough and the results overall were convincing that in general the proposed method outperformed baselines.

One reviewer found the experiments quite compelling while two found the setting where the methods works best somewhat unrealistic, i.e. when there is a uniform label distribution.  Indeed, in many benchmark datasets the classes are balanced but in practice the class distribution is rarely uniform.  In experiments where the label distribution isn't uniform, the proposed nuclear norm method does not outperform existing methods.  The authors are up-front about that limitation.   All reviewers remarked that this was a clear limitation of the work, but differed significantly in their assessment of how much that mattered.  

Other weaknesses raised by the reviewers were that baselines were limited and that novelty is somewhat low, particularly given that the nuclear norm has been explored in existing work before related to prediction confidence.  Another commonly listed weakness was missing baselines (Baek et al. 2022, projection norm, and other confidence-based measures).

Overall, all the reviewers listed ways they felt that the paper could be improved.  This along with the just below borderline average indicates that perhaps the paper isn't quite ready yet and could be stronger with some additional effort.
justification_for_why_not_higher_score: The reviewers all noted the limitation with respect to label imbalance and label shift, which seems to be the setting where this would be applicable in the real world.  In addition, the technical contribution is somewhat limited, i.e. nuclear norm of the prediction matrix and reviewers raised connections to novelty.  Thus it seems reasonable to expect some work / innovation addressing the label shift issue.
Decision: Reject