Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 3: Clear rejection
review_content: The paper propose to address the heteroscedastic regression problem using deep neural networks. It assumes the variance of heteroscedastic noise is known as privileged information and suggests to reweight the samples by their noise variance in the loss.

The major issue to me is the lack of novelty. Heteroscedastic regression is a classic problem in statistics. And reweighting using the inverse variance is a textbook method. See Chapter 10 of 

http://www.stat.cmu.edu/~cshalizi/ADAfaEPoV/ADAfaEPoV.pdf

This paper failed to cite any relevant reference and clarify the novelty.

To apply the method to a deep learning setting, some interesting problem can be how to estimate the variance with deep network in a reliable way (this was done previously using classic models). However, this paper did not tackle this harder (and more interesting) problem. Instead, it assume the variance is simply given during training. This is not very realistic in real world setting. The experiments are all synthetic and are not particularly convincing. 

Finally, the paper claims a lot of connection with privileged information (LUPI). But I found it hard to consider this variance a similar concept as privileged information, which is realistic and interesting.


review_title: Important References are Missing; Novelty is Unclear.

final_score: rating: 4: Ok but not good enough - rejection
review_content: In this paper, a reweighting technique is proposed to suppress the impact of heteroscedastic label noise in regression model training. The objective function of the regression model training process is composed of a weighted combination of instance-wise training loss. The instance-wise weight is determined by the estimated noise variance based on prior information of the label generation process. The weighting formulation is inspired by the best possible estimator of noisy measurements reaching the Cramer-Rao bound. 

The paper is clearly written. It explains well the problem definition and the methodological formulation. However, we think the innovation in this work is limited. The downside of this paper is as follows: 
1. It is a strong and usually impractical assumption to know a priori knowledge of label noise in the regression model training process. Especially, in the proposed method, the estimate of the noise variance needs to be accurate enough, so as to help suppress the noise impact accordingly. It is better to incorporate jointly the learning of noise distribution and the regression/classification 
model, so as to optimize the tolerance against the data-dependent noise. 
2. It only considers the noisy learning process for regression. However, in classification scenarios, noise (with respect to labels) is usually presented in the form of label flipping. The proposed reweighing technique is not directly applicable in that case. Please refer to the following work for further reading: 
Learning with Noisy Labels, Nagarajan Natarajan, Inderjit S. Dhillon, Pradeep Ravikumar, and Ambuj Tewari, NIPS 2013. 

review_title: This paper proposes a reweighed loss function for robust regression model training against label noise. The weight value of each training instance is determined by prior knowledge about the noise generation process. Results confirm empirically the merit of the algorithmic design

final_score: rating: 4: Ok but not good enough - rejection
review_content: ##########################################################################

Paper Summary:
This work targets regression tasks with noisy labels, and proposes to incorporate knowledge about the variance of the gaussian noise corrupting the observed labels to weight the loss function, at training time. The proposed method is evaluated in a series of experiments involving deep networks trained according to the weighted loss function, and compared to a baseline method that omits training samples that have a label noise variance larger than a threshold.
Results indicate the proposed method is more robust to noisy labels when compared to alternatives that do not exploit the information on the noise affecting labels.

##########################################################################

Reasons for score: 
On the one hand, the paper is extremely well written and somehow pedagogic, in that it provides compelling motivations for considering heteroscedasticity, and its possible sources, and general, intuitive descriptions of the proposed method before specialising them to the instance they evaluate. On the other hand, I think the prose lacks sufficient technical depth, on the model they use, its relation to “text book” material on heteroscedasticity, e.g. for Maximum Likelihood Estimation (MLE), and on the properties of the proposed method. The experimental evaluation, while representing a reasonable starting point, is not sufficient to fully understand the behaviour and the properties of the proposed method.
For these reasons, I think this work cannot be accepted as is.

##########################################################################

Positive points: 

1) The editorial quality of this paper is high, and the overall motivations given to support the problem statement are compelling and well discussed. This also relates to the fundamental assumption underlying this work: access to privileged information, taking the form of knowledge of the stochastic noise variance affecting observed labels, at training time.

2) The proposed method appears to be well positioned w.r.t. the recent literature on statistical modelling with noisy labels, especially concerning neural network based methods. It is unfortunate though that the literature scan doesn’t cover well-known approaches to tackle heteroscedastic noise in simple linear models, or in general MLE frameworks, which may be considered text-book material.

3) The experimental evaluation considers two regression tasks on their respective UCI/Bike sharing, and UTK Face datasets, considering several variants of noise generation processes, affecting labels in different and sufficiently realistic manner.

 
##########################################################################

Negative points:
1) My main concern is the “thin” contribution of this paper. The technical details of the proposed method are not sufficiently developed. Drawing inspiration from Fisher information calls for an appropriate discussion on the likelihood model, its noise model, to begin with. Then, I think the relation of the proposed idea to simple linear models, for which heteroscedastic regression has been studied in great detail (e.g. [1], for a general reference), and for MLE (e.g., [2]), would become more clear and would give the opportunity for the authors to develop what are the merits of their proposed method. For example, the weighted least square method is very similar to what is proposed in this paper. 
[1] Econometric analysis, Greene, William H, 2003, Pearson Education India
[2] Maximum Likelihood Estimators with Heteroscedastic Errors, G. R. Fisher, Review of the International Statistical Institute
Vol. 25,  1957
See also “Pattern Recognition and Machine Learning”, Bishop, 2006, (chapter 5 and 6)

2) The experimental evaluation is not sufficient to appreciate the virtues of the proposed method. For several noise distributions (including the additional ones considered in the supplement), the proposed method BIV does not seem to behave much better than the proposed baseline, that uses a simple threshold. Additionally, the figures are  cropped and do not allow to get a sense of what happens for all competing methods (Fig1 and Fig2). In Fig3, the figures report test loss and as there seems to be overfitting kicking in.
Also, it is mentioned the baseline method requires to set a cutoff parameter, but the proposed method also depends on a hyper parameter to optimise (done in the appendix). As a consequence, it is difficult to appreciate the main advantage of BIV w.r.t. the baseline.
Finally, in Fig.3, there are clear signs of overfitting: why did the authors suggest (end of Sec.3) that their work does not require regularisation?


#########################################################################

Main criticism:
I think, overall, the main criticism I have for this work is that the contribution is not sufficient. The main idea proposed in the paper fits sec 4.2, and it is based on well known results from text books. In eq.(5), the summation term is MLE with heteroscedasticity. The loss is scaled by a coefficient that collects statistics on the sample batch: if this batch is very small, or its elements not sufficiently diverse, I am afraid it could have a negative impact on the optimisation process (this is why, in the experiments, the authors chose a batch size of 256).
One possible way to overcome this criticism is to clarify the likelihood model, and compare the proposed method to existing approaches to address heteroscedastic Gaussian noise.
A possible advice would be to reduce (or move to the appendix) the discursive parts on heteroscedasticity, and the general formulations (e.g., sec. 2.2, sec. 4.1), and gain more space to explain how BIV is different from what is known.

Additional comments:
A note on experiments using the UTKFace dataset. In this case, the MLP used with 4 layers may be a bit “too simple”, in light of the high test loss on GT labels. Did you try with convolutional architectures, even simple ones?
Just to clarify, the test loss reported in the figures, as a function of training steps: what is the *test* batch size? Same as training batch size? The test loss is computer according to \sum_{k \in \text{test batch}} \mathcal{L}(f(\mathbf{x}_k, \theta), \tilde{y}_k) right?

review_title: Very well written paper, but I didn't find the contribution strong enough

Meta-review: The manuscript presents a deep network approach for heteroscedastic regression problem. It assumes the variance of heteroscedastic noise is known as privileged information and suggests to reweight the samples by their noise variance in the loss.

Three reviewers agreed that the manuscript is not ready for publication. The major issue is the lack of novelty. Heteroscedastic regression is a classic problem in statistics. And reweighting using the inverse variance is a textbook method. 

R2 and R4 confirmed that they have read author response. The rebuttals are useful to clarify some points, especially related to experimental settings and results. However, they are not convinced by the authors' argument on novelty and whether the assumption is realistic. 

Decision: Reject