Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: The authors propose a regularization approach for heteroskedastic data that regularizes in a data-dependent fashion, so different regions of the data spaces may be regularized differently by preferentially targeting high uncertainty and low density regions. This is achieved by estimating the noise level and density of each training example and then optimizing a regularized objective with input-dependent regularization.

How is k selected in (5)? Also, what is the purpose of Section 2 if neither the one-dimensional case nor the nonparametric model will be used in the experiments. The whole section seems distracting from the main message of the paper.

What is the definition of rare classes in the experiment in Table 1? it seems accuracy for the rare and noisy classes is calculated on all examples, including the 40% for which labels were exchanged. It would be interesting to see the accuracy also calculated for the correct (non-exchanged) labels, is that Table 4?.

Why are the test accuracies in Table 3 better than the validation accuracies in Table 2?

The results in Table 3 are certainly impressive, however, additional results in the Appendix are less so. Particularly, the experiment where HAR is compared to uniform regularization. It would be interesting to see similar results for WebVision. Also, why only exchange 5% of the negative samples but 40% of the positive labels in the IMDB experiment?

Minor:
- $f$, $l$ and $\lambda$ not defined in (1).
- $P(Y=y|X=x)$ in (2) should not be a function of $y$.
- why write the results for HAR and clean classes in bold?
- If the expectation in (4) is explicitly over the dataset why not simply write is as an average?
review_title: Paper 926 Review

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper proposed an adaptive regularization method to handle heteroskedastic and imbalanced datasets, which are closer to real-world large-scale settings. The framework applies a Lipschitz regularizer with varying regularization strength depending on the particular data point. The authors first theoretically study the optimal regularization strength on a one-dimensional binary classification task. By applying some simplification, the result can be extended to high-dimensional multi-class tasks and finally HAR algorithm is proposed. Experiments show that HAR achieves significant improvements over other noise-robust deep learning methods on simulated vision and language datasets with controllable degrees of data noise and data imbalance, as well as a real-world heteroskedastic and imbalanced dataset. The experiments show great improvement. However, since the derivations involve many approximations, the reliability needs to be confirmed by more experiments.

1. Assuming the pre-trained model is sufficiently accurate is not reasonable, especially in your complicated setting. 
2. This algorithm divides data into groups according to classes, and assume density and fisher information is constant within each group. This kind of division automatically takes class imbalance into account, which is not necessary according to your theory. Since heteroskedasticity means uncertainty varies between instances, not between classes, what if we divide data into groups with equal size? 
3. Synthetic experiments should be conducted to prove at least two things: (1) your algorithm can correctly estimate the regularization strength. (2) your algorithm can be applied to real heteroskedastic datasets, which vary uncertainty between instances, not classes. 
4. The imbalanced (long-tail) experiments did not compare with current SOTAs, i.e., BBN (CVPR20).
5. You should compare your method to other methods with similar ideas, i.e., MetaReg.
review_title: A novel task for real-world long-tail learning

final_score: rating: 7: Good paper, accept
review_content: **Summary.** This paper presented a novel data-adaptive regularization scheme to adjust for the heteroskedasticity and non-uniformity (i.e., imbalance) of data distribution. Heuristically, solutions are subjected to heavier penalties in regions with either higher aleatoric (large variance) and epistemic (low-sample) uncertainties. While the exact proposal is, in theory, computationally intractable, the author(s) have made some clever relaxations, which lead to a very practical solution.  I very much enjoyed reading this paper. 

**Quality & Clarity.** This paper is well organized and clearly written. The. author(s) start the discussion with very intuitive examples,  followed by rigorous mathematical development on how to derive a theoretically justified adaptive penalty. To reach practical solutions, relaxations and surrogates are carefully elaborated. The experiment section is also well-executed, covering convincing synthetic and real-world examples to demonstrate the effectiveness of this proposal, comprehensively compared to SOTA alternatives. 

**Originality & Significance.** Although the idea of developing uncertainty-aware models have been repetitively explored in literature, I do find the HAR model proposed in this submission fresh & appealing. One of the main novelty that I appreciate is the fact the author(s) have scaled the gradient penalty for model complexity to deep neural nets, which in the final solution is replaced by the Lipschitz estimate. Although the stratification of sample space feels a bit hacky, I believe that is a necessary compromise to be made. I am reasonably positive this paper is expected to make some impact. 

**Minor issues.** There is a missing bracket in the last equation on pp 3. Eqn (6) should perhaps be more explicit on the dependence for $f$. I would love to see discussions, preferably preliminary experiments, to explore the use of nonparametric density estimation schemes to replace sample stratification. 

review_title: A nice paper

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: The authors propose a novel regularization approach aimed at addressing issues of class imbalance and heteroskedasticity. This adaptive approach uses a Lipschitz regularizer with varying strength in different parts of the input space, regularizing harder in cases of rare and noisy examples. The authors derive the optional regularization strength in the one-dimensional setting, to set ground for the proposed approach and its application in higher-dimensional settings. The approach is evaluated on multiple image datasets, and a textual dataset - and compared to a number of baselines, including those involving noise-cleaning, reweigthing-based methods, meta learning, robust loss functions, as well as tuned uniform regularization. The improvements seem quite strong, and clearly demonstrate the utility of the proposed approach. The paper is well structured, clearly written - and was a pleasure to read.

I only have brief comments:

- Unless I am mistaken (I could have missed it), I didn’t see a concrete mention of a statistical test that was used to determine statistical significance. The authors do use the word ‘significant’ to qualify the observed differences, but that should only be phrased as such if appropriate statistical testing has been done and has been outlined when describing the evaluation. That being said, the improvements are quite stark and I don’t doubt the validity of the claims - I am merely suggesting that the authors should include this information for clarity.

- While the evaluation clearly shows the benefits of the proposed approach, and there is a detailed ablation study, I think that it would be interesting to additionally discuss and analyze in more detail the resulting regularization coming from the proposed approach, on WebVision (or one of the other datasets, wherever it’s easiest) - to show how the distribution of how many examples are being regularized by which factor - and contrast that with the optimal uniform regularization. There could also be a correspondence with Figure 1, if constraining the comparison to freq,rare ; noisy, clean input types. Visualizing some such distributions would help understand the method better - and add to the presented analysis.
review_title: Excellent paper, good improvements

Meta-review: This paper got 2 clear acceptance and 2 borderline recommendation. The main concerns lie in the clarity of the experiment results and settings (AR3). The authors address these questions in their response. AR2 has two important questions. One is whether the simplified assumption holds in the considered very complicated settings (i.e., the labels are noisy and long-tailed). The other one is the lack of comparison with SOTA method for long-tailed classification. The authors did good job in their response. They provide additional experiment results to address these questions. Overall, the quality of this submission meet the bar of ICLR acceptance, though AC has concerns on the complicated settings and the marginal performance improvement over the existing long-tailed works.
Decision: Accept (Poster)