Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: The authors propose a new framework to evaluate the generalizability of trained networks. The framework is based on evaluating the perturbation response curve on intra and inter-class sample mixup. It uses two novel statistical measures: the Gi-score and the Pal-score to predict the generalization gap. The framework is able to achieve a state-of-the-art predictive measure for several tasks in the PGDL competition and can be used to capture invariants to parametric input transformations.
main_review: Originality:
While closely resembling [1, 2], the proposed method includes some substantially novel and original contributions including inter-class sample mixup as well as a spectrum of parameterized perturbations.

Quality:
The paper successfully supports the predictive generalization claim through experiments based on the PGDL competition dataset and the predictive invariance claim based on a custom dataset built around the CIFAR-10 and SVHN datasets. It would have been additionally beneficial for the authors to add experiments with more challenging datasets like [3, 4] which were built to evaluate the generalizability and robustness of the networks to domain shift.

Clarity:
The paper is well written and clear.

Significance:
The paper shows significant improvement to the predictive generalizability problem which is very important for the goal of making better, more generalizable models. Also, the framework is successfully able to learn task-dependent invariances, which is a critical task for many machine learning applications. One potential limitation of this framework is that it has not been validated on non-image modalities.

References:
- [1] Natekar, P., & Sharma, M. (2020). Representation based complexity measures for predicting generalization in deep learning. arXiv preprint arXiv:2012.02775.
- [2] Kashyap, D., & Subramanyam, N. (2021). Robustness to Augmentations as a Generalization metric. arXiv preprint arXiv:2101.06459.
- [3] Barbu, A., Mayo, D., Alverio, J., Luo, W., Wang, C., Gutfreund, D., ... & Katz, B. (2019).  Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models. In Advances in Neural Information Processing Systems 32, pages 9448–9458. 2019.
- [4] Koh, P. W., Sagawa, S., Xie, S. M., Zhang, M., Balsubramani, A., Hu, W., ... & Liang, P. (2021, July). Wilds: A benchmark of in-the-wild distribution shifts. In International Conference on Machine Learning (pp. 5637-5664). PMLR.
limitations_and_societal_impact: The authors have adequately addressed the limitations of their work.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The paper proposed two scores calculated from accuracy changes at varying levels of perturbation in training samples for predicting neural network generalization. The scores, Gi-score and Pal-score are inspired by the Gini and Palma index from economics.

main_review: The authors proposed a new framework, starting with evaluating trained networks on a subset of training data with varying levels of perturbation. They then construct the perturbation response (PR) curve and compute the Gi-score and Pal-score based on the PR curve. Two sets of experiments are done to demonstrate the usefulness of the scores (mainly Gi-score) in predicting neural network generalization and measuring network invariance. 

The generalization prediction experiments are based on the PGDL competition, and in some cases, the performance of the proposed method surpasses the winning submission. Combinations of perturbation method (intra vs inter) and perturbed layer (input vs layer1) were used. The best performance was achieved with different combinations for each dataset. This seems to make the framework difficult to implement -- do people need to run all combinations (or even more) to reach a conclusion? Moreover, Gi-score almost always outperforms Pal-score. Is Pal-score more useful in any scenario? Although the authors draw connections with Gini coefficient and Palma ratio, they did not provide more intuition as to why these measures may be useful, or in what circumstances one measure is preferred to the other.  More discussions on this will help readers better appreciate the proposed framework. 

In the generalization prediction experiments Gi-score is better than the value halfway on the PR curve (Mixup accuracy is $A_{0.5}^{(0)}$). For measuring invariance, Gini score is shown to be better than the mean accuracy and ($A_1^{(0)}$) at max perturbation. Again, more intuition and discussion on why Gi-score is a better choice will be helpful. 

For presentation, while I appreciate it that the authors included pseudocode for Gini-score calculation, I’m afraid it does not provide more information than the shaded area in Fig.1. In fact I was surprised that Alg.2 was needed and had to read it in case I misunderstood the method. The space could be dedicated to more discussion and intuition.  I have a similar feeling about Alg.1.

The authors mentioned the lack of theoretical results a limitation of the paper. Indeed, the paper needs more intuition and discussion for readers to better appreciate the proposed framework. 

limitations_and_societal_impact: The authors described the limitations.
Societal impact: N/A.
review_title: null

final_score: rating: 3: Clear rejection
review_content: summary: This paper firstly uses Gini coefficient and Palma ratio together with PR curve for measuring generalization. The paper claims that their framework outperforms the state-of-the-art measures on most of the tasks in the PGDL competition and can capture invariance to input transformation for a trained network. Finally, the paper shows some experimental results on image classification.
main_review: This work is about predicting deep neural network generalization but the authors focused too much on invariance. Maybe the authors should explain more about the relationship between generalization and invariance. They firstly use Gini and Palma statistics to measure generalization, which is novel, however, their usage is humble.
limitations_and_societal_impact: The usage of Gini and Palma statistics in the paper lacks analysis and explanation (e.g., which one to use and how the parameter should be set). Experimental results show that each type of their measure could perform much better even though some of them could be extremely bad. This phenomenon is not explained in this paper.
review_title: null

Meta-review: The paper presents the idea of perturbation response curve and by combining that Gi-score and the Pal-score, proposes a new way for predicting the generalization gap. The method is able to achieve impressive score on several tasks in the PGDL competition. Reviewer wkSZ provides a strong support and finds the submission well-written and the result significant. Reviewer r6eX give a borderline rating. One of the concerns of this reviewer was related to the requirement of having different hyperparameters for the proposed scheme for each dataset, in order to achieve the best prediction result. In the rebuttal, the authors performed additional analysis and reported a common setting that performs well on most of the PGDL tasks. In addition, reviewer r6eX about the intuition behind Gini coefficient and Palma ratio and the reason they are useful, which is again explained by the authors in their response, and planned to be included in the final revision. Reviewer fdZt had stronger reservation against the paper, and was mainly seeking more theoretical justification for the proposed scheme. Overall, the reviews were diverge, but I myself lean to accept the submission, because I think the paper excels in the empirical side (on various PGDL competition tasks) and it is interesting enough of a result without need for a theory component, and the findings of this work can provide new insights about generalization, and be a stepping-stone for deeper analysis in the future. I recommend accept and ask authors to incorporate the clarifications/explanations that provided here in the paper as well.

Decision: Accept (Poster)