Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: ##########################################################################
Summary:
 
The manuscript identify robustness properties of the f-divergence optimization when noisy labels presents. The f-divergence measure is defined between the joint distribution and the marginal distributions of classifier’s predictions and labels. In this case, the f-divergence measure can be interpreted as the f-mutual information and thus the optimal classifier can be estimated by maximizing the f-divergence measure. The authors examine a family of f-divergence functions (generators in f-divergence measures) and derive several interesting robustness properties. For example, total variation is robust for the label noise problem (the label noise does not affect the optimality of the maximizer; it does not require the knowledge of noise rate), and when the f-divergence functions are not robust, we can still correct the solution to be the optimal by calculating the bias term introduced by the noisy labels.
   
##########################################################################
Reasons for score: 
 
I vote for accepting. The manuscript provides theoretical analyses of the f-divergence optimization with noisy labels. Although there are several research works on training with f-divergence measures recently, I can see the novelty of this work regarding the robustness of the f-divergence optimization when noisy labels presents. The effectiveness of optimizing f-divergence with label noise is also verified by a set of numerical experiments. I think that this work can be a good reference about robust training approaches dealing with label noise. 
 
##########################################################################
Suggestions 
I have just few minor suggestions.
1)	At first glance, maximization of f-divergence measures sounds confusing in the abstract and introduction as we generally minimize any divergence measures to train a model to fit the training data. I think that it would be better if the authors include additional explanations (in the abstract and introduction) that the f-divergence measure can be understood as the f-mutual information in this problem formulation.
2)	Below equation 2), f(v)=log v should be f(v)=v log v   

review_title: good paper. I have just few minor comments. 

final_score: rating: 6: Marginally above acceptance threshold
review_content: The authors consider the problem of learning a classifier when the labels are noisy. They consider an approach where they optimize the mutual information between the predicted label and the noisy label (or more generally, a variant of mutual information defined by an f-divergence). To optimize over mutual information, they use the variational form of f-divergence. They show that their loss function is a biased version of the mutual information between the predictions and the clean labels. They argue that the bias term is small and that, for many choices of f-divergence, that the true optimal solution is optimal even in the noisy case over a specific subset. They then experimentally demonstrate that their approach performs better than the baselines for a large number of datasets and noise settings.

I found the paper interesting, however the theoretical justification is not fully clear to me. The authors should clarify the following two questions.

(1) Is f-mutual information a good loss function on clean data, for any choices of f-divergence? I see in Section 2.2 special cases for when this is true, but can f-mutual information return a very bad solution when these conditions are not true?

(2) In Theorem 8, why is $\mathcal{H}^*$ the only set that we need to worry about?

The authors use the variational form of f-divergence to optimize their loss function. Algorithmically, how is the term $g^*$ calculated and updated? Since $P$ and $Q$ are changing as the model is optimized, does $g^*$ need to be recalculated over time?

The experimental section shows compelling results compared to state-of-the-art approaches. I think including no noise as an additional comparison can improve the understanding of the theoretical questions from above.

Overall, I found the paper interesting with solid experimental results, though I still have some questions of the theoretical principles of the work.
review_title: Interesting work with solid experiments but some theoretical questions

final_score: rating: 7: Good paper, accept
review_content: This studies when maximizing a proper f divergence measure between the predicted labels and the observed labels would make the model robust with label noise. The authors derive that the f divergence can be decoupled into variational differences defined on distributions without noise and a bias term introduced by the noise. With this measure, we can learn with noisy labels without knowing the noise rate. They also consider the situations when the proposed f divergence measure is not robust against noise in labels, they propose a correction to mitigate this issue. Comprehensive experiments are done across several datasets with three noise models. Results show the proposed methods outperform baselines in most cases.

If the authors could explain the assumptions on the label noise that the work is based on early in the introduction and Section 2, the presentation would be improved. Now I have to read until Section 1.1 and Section 2.3 to know noise labels are generated based on the transition matrix.

In Table 2, in the sparse and high noise situation, the PL model outperforms the proposed method consistently. It would be better if the authors can discuss the difference between PL and the proposed method and provide reasons about why PL is better. Currently, the discussion focuses on why the proposed method cannot do well. In addition, the current explanations are conjectures. I would suggest the authors run experiments to verify such conjectures. For example, analyze the input of the g function to verify the first conjecture.

In Table 2, there are no results on random noise for MNIST and fashion MNIST. In addition, it did not distinguish Sparse Low and High and Uniform Low and High for CIFAR 100, could the authors explain this?

In Table 3, the J-S model with sparse high noise produces a gap value -20.88, which is way larger in magnitude than others. Could the authors further explain this result? It also lacks discussion on the meaning of the gap value, I wonder why such a large negative value has a trivial impact on the performance of the model. 




review_title: Good paper. A solid formulation and analysis of learning with noise labels by optimizing f divergence between predictions and labels. Includes both theoretical contribution and comprehensive empirical results.

final_score: rating: 7: Good paper, accept
review_content: The authors study optimising f-divergence for supervised learning in the context of label noise. They show robustness to noise for certain families of f-divergence functions and devise a technique for minimising bias for other f-divergence functions. They evaluate their method against pre-existing methods as baselines on a variety of learning tasks including simulation and real-world data. There are some minor issues to address but overall a good submission.

# pros

- Easy to read paper, the authors guide the reader well through the theory
- proofs are sound
- good set of experiments
- results are interesting and applicable in many domains, of broad interest to the community

# cons

- The experiments are not described in sufficient detail. It is unclear what h is used for experiments and what variational function was used exactly. The authors state g* can be updated progressively or fixed beforehand, but neglect to say what they did for experiments. 
- The authors mention the BLC and FLC methods require estimating the noise transition matrix: how was this done?

# minor things

- typo on page 3: f(v)= log v for KL divergence
review_title: f-divergence is robust to label noise

Meta-review: The paper tackles a very important problem. The formulation of the paper is sound as under lightweight assumptions, the supervised loss follows an f-divergence formulation (see "Information, Divergence and Risk for Binary Experiments" by Reid and Williamson (JMLR 2011), in particular Section 4.7). It would make sense to dig in the loss in the context of label noise; the variational formulation provides an interesting direction along those lines. The rebuttal on the experimental concerns of reviewers is appreciated (Cf authors’ rebuttal summary).

Decision: Accept (Poster)