IS THE LABEL TRUSTFUL: TRAINING BETTER DEEP LEARNING MODEL VIA UNCERTAINTY MINING NET
Authors: Yang Sun
Abhishek Kolagunda
Steven Eliuk
Xiaolong Wang
Venue:  (2020)
Keywords: Semi-supervised Learning
Robust Learning
Deep Generative Model

Abstract: In this work, we consider a new problem of training deep neural network on partially labeled data with label noise.  As far as we know, 
there have been very few efforts to tackle such problems.
We present a novel end-to-end deep generative pipeline for improving classifier performance when dealing with such data problems.  We call it 
Uncertainty Mining Net (UMN).  
 During the training stage, we utilize all the available data (labeled and unlabeled) to train the classifier via a semi-supervised generative framework. 
 During training, UMN estimates the uncertainly of the labels’ to focus on clean data for  learning. More precisely, UMN applies the sample-wise label uncertainty estimation scheme. 
 Extensive experiments and comparisons against state-of-the-art methods on several popular benchmark datasets demonstrate that UMN can reduce the effects of label noise and significantly improve classifier performance.

Under review as a conference paper at ICLR 2020
Is THE LABEL TRUSTFUL: TRAINING BETTER DEEP LEARNING MODEL VIA UNCERTAINTY MINING NET
Anonymous authors Paper under double-blind review
ABSTRACT
In this work, we consider new problem of training deep neural network on partially labeled data with label noise. As far as we know, there have been very few efforts tackle such problems. We present novel end-to-end deep generative pipeline for improving classifier performance when dealing with such data problems. We call it Uncertainty Mining Net (UMN). During the training stage, we utilize all the available data (labeled and unlabeled) to train the classifier via semi- supervise generative framework. During training, UMN estimates the uncertainly of the labels' to focus on clean data for learning. More precisely, UMN applies the sample- wise label uncertainty estimation scheme. Extensive experiments and comparisons against state-of-the-an aht methods on several popular benchmark datasets demonstrate that UMN can reduce the effects of label noise and significantly improve classifier performance.
INTRODUCTION
Deep Learning (DL), to learn powerful representations, it usually requires a large amount of training data. However, for many real world problems, it not always possible to obtain sufficiently large training data. What we can usually get is limited training data with corrupted labels which heavily affect the model performance. Although acquiring large data is not hard, considering the information explosion on the internet, accurate labeling is usually an expensive and error-prone task which involves humans' interaction, especially experts with knowledge in the specific field. Most of the time, we have to build DL models using limited training data with corrupted data labels. It becomes very challenging to apply current popular deep learning frameworks to solve this problem.
Training a deep learning model via noisily labeled data is challenging task (Patrini et al.) 2017
Li et al. 2017 Ding et al. 2018, Menon et al. 2015 Goldberger & Ben-Reuven, 2016 Jiang et al. 2017). Most of current explorations for dealing with such problems usually require large amount of labeled data as a prerequisite. means that in these works, the designed pipelines and evaluations are based on having enough labeled training data, though they may include label noise. These approaches will suffer from the limited size of labeled training data as well as the label noise. As we know, the hierarchical representation learned by deep learning models mainly benefits from the amount of data. The deep learning model can easily overfit to the mislabeled samples especially when the data size is small (Tarvainen & Valpola 2017 Ren et al. 2018).
Unfortunately, there is no general solutions to solve the problem of training on limited annotated data with label noise. If we directly train the deep learning model only with the labeled data, the small portion of labeled training data will limit the learning capability. Meanwhile, the label noise makes very challenging utilize the labeled data directly. In this work, our framework can handle these problems simultaneously. UMN includes two major components. The first part is used to learn a latent feature representation via a generative framework (VAE) using all the training data. Then the learned embedding is applied the input to its subsequent semi-supervised learning component by conditional variational autoencoder to train the classifier. To handle the label noise, we integrate label uncertainty estimation module where the reliable data contributes more the model training and noisy data contributes less.
Though recent works (Hataya & Nakayama 2019 Ren et al.) 2018) discuss such problems, they usually assume to have a separate small clean dataset to begin with. It may not be possible to have an independent clean dataset in real scenarios. Current supervised robust learning approaches cannot

---

Under review as a conference paper at ICLR 2020
be directly adopted to our case since the labeled data is small and not reliable. The limited labeled training data makes the supervised model overfit very easily to the small size of noisy data. This will heavily hurt the model performance. The limited data labels and noise can also affect other related state-of-the-art semi-supervised learning approaches. We will present the comparisons with these approaches in the experiment section.
In this paper, our major contributions can be summarized into the following aspects: First, our studied problem is very general in real scenarios: small mislabeled datasets with large amount of unlabeled data, and there are very few existing approaches to target this kind of problems. Second, instead of treating the label noise as the preknowledge, we explicitly use the deep generative architecture to model the noisy labels at the sample level precisely. This has been validated by the improved experimental results as well as the theoretical analysis. Third, we show how to use the moving average model to estimate the sample-wist wise uncertainty in labels. We have explained it in the theory analysis section and validated this assumption in the experiments. Furthermore, our experiments demonstrate that UMN can also help identify these mis-labeled data without any prior knowledge. Finally, we build one end-to-end deep learning pipeline to train against the noisy labeled data.
2
RELATED WORKS
Along with the popularity of deep neural network, the problem of training deep neural networks with noisy label data has started to draw our attention. Natarajan et al. (Natarajan et al. (013) propose the unbiased estimator of the surrogate loss function and calculate theoretical bounds for empirical risk optimization. Modeling the consistency as the regularization for deep neural network is another route for robust learning with noisy labeled data. Reed et al. (Reed et al.) 2015) design a robust loss to model the prediction consistency. In (Li al. 2017). the authors propose knowledge distillation framework where they train an auxiliary model on a small set of clean data samples and lincarly combine its predictions with the observed labels to form the new targets to train. In (Ding et al. 2018) a two-stage approach is proposed. First a DNN model is trained on the noisy data and used to filter out these noisy samples. Then another model is trained on the filtered dataset using semi-supervised approach by treating these filtered out samples as unlabeled. Menon et al. model the corruption process of dataset by learning the class-probabiling estimator (Menon et al., (015). Another way of regularization for training on noisy labels is to estimate the class conditional corruption ratio. Goldberger et al. (Goldberger & Ren-Reuven] 2016) propose a softmax layer along with the classifier to predict the class conditional corruption ratio. In (Jiang et al. (017)), they propose MentorNet which learns a data driven dynamic curriculum to be followed by the StudentNet.
Semi-supervised learning with few labeled samples is also well studied area. Some of the closely related works are (Gordon & Sormanddzz-Lobtto 2017). (Kingma et al.] 2014). (Tarvainen & Valpola 2017). Kingma et al (Kingma et al. (214) propose stacked deep generative semi-supervised model for training on partially labeled datasets. They first train a variational autoencoder on the labeled data, then stack on top of it conditional variational autoencoder and continue training in semi-supervised fashion to get robust classifier. In (Tarvainen & Valpola 2017). the authors train classifier in semi- supervisdd way using temporal ensemble approach. They train a student network and maintain a teacher network as the exponential moving average of the student. They use classification loss for labeled data and enforce consistency loss between teacher and student for unlabeled data.
There have been limited works dealing with bi-quality data (Hataya & Nakayama] 2019). (Langevin et al. 2018). Bi-quality data is defined in (Hataya & Nakayama (219)) as datasets with few labeled samples where the labels are potentially corrupted. In (Langevin et al. 2018). the authors discuss to use the deep generative semi-supervised model to model noisy labels. They assume that the labels also randomly corrupted with probability p. Their idea relies the predefined p train the classifier. However, the corruption probability P is not known as a prior in reality.
Compared to these work, we present new architecture for training a robust classifier with dataset including noisy labels. We assume that there is no preknowledge of the label corruption rate and we do not have access separate clean dataset which is more close to the real scenario.
2

---

Under review as a conference paper at ICLR 2020
gradient
Cossiiicatoon Cost
Uncertainty Estimate
Predt
VAE
Casiifier
C-VAE
Classifer
Barcan Cost
cool
FMA Update
reights
0'
X
Figure I: An overview of UMN framework: X and are the observed inputs and potentially corrupt labels. 0 and O' are semi-supervised generative models. 7 and r/' are noise functions that perturb the input X. The red and green colored arrows represent the flow unlabeled and labeled data through the models. A is updated via stochastic gradient descent approach and VAE losses, and 0' is the Exponential Moving Average (EMA) of 0. The sample-wise uncertainty (e) between the predictions of 0 and I' are used to re-weigh the gradients that back propagated to 0 via the classification loss. The general architecture for 0 is adapted from (Kingma et al. 2014). VAE indicates Variational AutoEncoder and C-VAE represents Conditional- VAE. Note that in O' when making predictions (on labeled data) only the weights from encoder of the VAE and the classifier are used. For simplicity, the KL. loss for the predictions on unlabeled data with respect to an uniform prior is not shown. For simplicity, we omit the optional consistency loss term between the classifiers of and 0' for unlabeled data the figure.
to minimize the classification
3 METHOD
The goal of our work is to learn a better model with the limited labeled data including the noise. Since the labeled data only occupies a small portion of the training data, the labeled data alone not sufficient to train good model. At the same time, we also need to solve the label noise problem. To deal with these issues, our pipeline consists of two major components. In the first part, we apply unsupervised learning scheme to learn latent feature representation via all the available training data. Then we utilize the label via semi-supervised learning pipeline to train the classifier. At the same time, incorporate the label uncertainty estimation to reduce the influences coming from these mislabeled data. We estimate the uncertainty of the given sample from the prediction of the updating model and its guider model (exponential moving average of the updating model), part of this idea is inspired by work (Polyak & Juditsky 1992). The estimated uncertainty is then used to assign weights the training data samples. This an end-to-end framework. We call Uncertainty Mining Nets(UMN), In the following subsection, we will introduce UMN in more details.
3.1 LATENT-FEATURE LEARNING
In this work, we represent the given data as the set of (X,Y) {(11, 9/1 y..,, (IN, yN)} where Jy is the observed data sample and is its corresponding label which may be potentially corrupted. If I; is unlabeled, then y/4 becomes empty. To be concise, we omit the index in the rest the paper.
Motivated by the recent success of generative models in semi-supervised learning related applications, firstly, we apply the variational autoencoder (VAE) on all the training data. VAE consists of two modules: an Encoder that maps the variables z to the latent variables approximate the prior distribution of p(z) and a Generator po (x|2)) that samples the the input variables æ given the latent variables z. It can be formulated as
p(z) (o 2,;;
3

---

Under review as a conference paper at ICLR 2020
where f2; 8) is the likelihood function and A indicates the Gaussian distribution. As are network parameters. The goal of VAE is to maximize the evidence variational lower bound (ELBO) of pa as (x) > [ log po [z)2) - 1 (gg (z]z))lp(z).
The learned latent variables obtained from VAE can be used as the feature representation to train the classifier. Based on this, we can fully utilize the knowledge from the training data without the constraint of the labeled data size limitation. The low-dimensional embedding & can well cover the distribution of the input data æ. Our hypothesis is that the unlabeled data helps training robustly against the noisy labels via the deep generative modeling. We will see this in the next subsection.
3.2 SEMI-SUPERVISED LEARNING
After building the unsupervised learning framework via VAE, we apply the label information to improve the learning capability of the neural network. Instead of using the raw data z, we directly use the latent vector 2, obtained from VAE of letuut- learning part along with the pre-defined labels. We follow the idea of conditional generative model.
Our generative process can be formulated as,
y, Mv), p(za). Zn po(=a|=0.V) v),
x ~ po (z]|za), 1 ~ po po y; za);
(3)
where is the multinomial distribution. y indicates the true label. &. is jointly generated from &s and y and 25 is the latent representation of the input. In this work, we treat y as the latent variables. We use ý to denote the observed labels. To more accurately capture the reliability of labels, we estimate each labeled sample uncertainty via po(ū (g) y. 2a). the posterior distribution can be factorised as
(Faa =b, u| 1) ((=a|>)) (z) (vyl=a) =a) g(2b) Ra: v)-
Following this, the Evidence Lower Bound (ELBO) can be derived follows:
Σ
(=,
Σ (llel
p(z)))
xE(L,U}
DEC
Σ
Σ (3122 KL p(v))
qo (zul
xE(L,U)
DEC
Σ
E (xul
Σ
go 2. ( BE
loog po (za 20. v) log loggo(>alx))
xE(LUU)
NEC
Σ
E (xalk))) logpo(x(=a) +
Σ (xa
Σ
(o =a log pol za),
(5)
xE(LU)
xE
BLEC
where (v(yy2 z.),, (s |z), go(22/ v), pe pz(Zzlzz (26,))))).)))) po (z]2a) indicate the classifier, encoder, condi- tional encoder, conditional decoder and decoder functions respectively. indicates labeled data, U means unlabeled data and C denotes the set of used classes. The last term can be treated as labeled loss term as the supervised portion of Eq.
For more detailed derivation of the ELBO, please refer to the Appendix. As indicated in Eq. 5 except the last term, all other terms in the equation are calculated over all training data samples. The last term serves as the supervised factor for semi-supervised learning where the observed labels may be incorrect. It uses an uncertainty related function po (ÿ Wk 2a) to modulate the sample weights during the gradient back- poopppropagttion. As indicated in Fig. pa ( (yly), zo) is approximated based on the sample-wise uncertainty e. More details are given in the following.
For simplicity, we can also represent the generative process of the observed ý by Fig.
One plausible way to estimate the uncertainty of the sample label to model the class conditional corruption ratio p y), which is mean field approximation. As discussed in (Langevin et al. 2018 the class conditional probability can be computed as
11-E p(lv)) = 1/(C-
if = 1), otherwise
(6)
11,
4

---

Under review as a conference paper at ICLR 2020
Figure 2: The graphical model generating the observed label
and the labeled loss term can be formulated as
E (a ((l)a)
where f(e) logl أ
function with constant e. From Eq. 7 can see that it assigns the same weight for all samples belonging to the same category, i.c.. all labeled terms contribute equally to the gradient back propagated regardless of their label correctness as
(eo(e.l ((c)Vogo(vle
Although this idea is simple and easy to follow, the assumption may not be the case in the real scenario. In general, the value of which the class conditional corruption rate, remains unknown beforehand. Another issue is that it's not accurate to apply the same categorical corruption ratio to all data samples of a given class. Our approach solves these problems by estimating sample-wise uncertainty during the training stage.
We replace the class conditional in Eq. 9 with point-wise estimation the label uncertainty c(za). where we follow the generative process of Eq. The uncertainty of label is hence e(za). Pointwise label uncertainty is calculated as
(1-c(20),
if i = c((2)/(C' - 1). otherwise
p(1 1,, nn)
So f(c) in Eq. becomes f(e) = logl (C-1)[1-((ao That is, samples contribute relatively more to c(z.)
the gradient backpropagation when the estimated e is small which indicates there is a high probability that the given label is correct. By focusing more robust classifier. From this, we can see that the key is to approximate (2za) precisely. In the following subsection, discuss the proposed approach to model c(2a) using the differences in predictions from the updating classifier model and its moving average model.
these reliable targets during training, we can train
We want to emphasize that although UMN is based VAE framework, its architecture can be flexible. Depending on the task, the encoder of its generative model-VAR can be implemented using various deep learning networks, e.g., AlexNet, GoogLeNet, ResNet, etc. Meanwhile, UMN is not limited to one specific task, like image classification. It can be casily extended to other classification problems.
3.2.1 UNCERTAINTY ESTIMATION
As discussed above, we aim to train more robust classifier in an end-to-end semi-supervised architecture. The key to this idea is to estimate sample-wise label noise (e(=a)) this cannot be known as a priori. We are motivated by the initial work of the iterate average model (Polyak & Juditsky [1922) where exponential moving average of the model weights can be used to stabilize the training in stochastic gradient based approaches. Exponential moving average of the model weights can be calculated as:
8, 7 (1-
(10)
where 0 is the set of weight parameters of the classifier model and 6' is its exponential moving average, 7 controls the smoothness of model updates. We name them Learner and Guider models respectively. I is the step index of the iterative optimization. Since this iterate average gives optimal bound for convergence rate (Polyak & Juditsky 1992) and can be less sensitive to the noisy updates (Eq. [0), we adopt it as the Guider model to estimate the label uncertainty.
We approximate the label uncertainty via the absolute difference in the predicted probability of the Learner and the Guider for the observed class as following
(Zz,) (2()) f(zx) I,
(11)
5

---

Under review as a conference paper at ICLR 2020
where f' and indicate the classifier's predictions of the learner and guider's respectively. As the Learner learns from the noisy labeled data, the Guider model is used to weigh the samples based on uncertainty thus limiting the contributions of unreliable samples to the gradients back propagated. In the following subsection, we provide the analysis of the reason why using the difference between the predictions of the Guider and Learner models is good measure for the uncertainty estimation.
3.2.2 THEORETICAL ANALYSIS
Our deep generative model allows to model the correct labels via the observed labels and therefore leads to a plausible way to adjust sample weights during training. We will show the following theorem that the label correctness is associated with the difference between the Learner and Guider models during training.
Theorem 1. Suppose that 0* the optimal solution of the optimization problem. Or, is the stochastic random variable for SGD (learner) and fG is iterate average (guider) of Oz. (a -rr
+ Sh (t)t),t. The distance between Or, and O* is consistent with the distance between Oz and 0g in the sense that
8 ,
E
|10 0011)) (0z (0z- -06]] >0
(12)
which holds when the training epoch T satisfies the following relation
1 Tr (H-
T X [E
(13)
where X is the constant learning rate.
We can see from Eq 12, during the early training stage, the distance between the learner and the optimal solution increases when the distance between the learner and guider increases and vice versa. This leads to preventing samples with uncertain labels to participate in training and thus reducing the overfitting on potentially wrong labels. The proof of the theorem given in the Appendix.
EXPERIMENTS AND RESULTS
Our experiments are designed evaluate whether UMN is effective approach learn good model with limited annotated training data which include label noise. We compare to popular approaches of supervised learning, supervsseee learning and robust learning dealing with noisy labels. These methods represent state-of-the-a approaches for model learning with noisy data. Moreover, we also compare the results with the pipeline without using our uncertainty estimation module. Further, we also evaluate the performance of applying the uncertainty estimation module in identifying corrupted labels.
We experiment on variety of image classification problems with varying degrees of label cor- ruption rates. We use three popular datasets including MNIST, SVHN and CIFAR-10. For a comprehensive evaluation, set up five different uniform labels corruption rates including [10%, 20%, 30%, 50%).
4.1 IMPLEMENTATION DETAILS
In the experiments. supervised deep learning framework means all the label data are directly used for training. For SVHN and CIFAR-10, we experiment a CNN architecture with 13 convolutional neural network as well as the ResNet-101 architecture (He et al. 2015) as baselines for supervised learning approach. To compare with semi-supervised learning approaches, we select two recent popular works. One is named as mean-teacher (MT) proposed by Tarvainen and Valpola Tarvainen Valpola 2017) and the other one is Langevin al. (Langevin al.) 2018) named Mislabeled VAE (M-VAE). Langevin et al. (Langevin et al. 2018 only roughly discusses their idea without showing the details of the model architecture and experimental results popular benchmarks. In this work, we implement the idea in (Langevin et al. 2018) and compare with UMN in all three different datasets. In our experiments, the encoder and decoder architectures for UMN and M-VAE.

---

Under review as a conference paper at ICLR 2020
In comparison to robust learning approaches, apply another two recent works including Mentor- Net (Jiang et al. 2017) and Reweight (Ren et al., 2018). For a fair comparison with these works, we follow all the onginal setting in these papers
experiments. For each corruption ratio, ran 5 experiments by randomly choosing samples to corrupt each time, and report the mean error rate.
As illustrated in Fig. D our framework (UMN) is composed of two encoders and two decoders (VAE and C- E) along with a classifier. For the experiments in MNIST, We adapt a multi-layer perceptron (MLP) architecture similar the one proposed in (Kingma et al.) 2014), for the VAE, C-VAE and the classifier models. The encoders, decoders and the classifier of our model use a single hidden layer with ReLU activation and an output layer with no activation. For SVHN and CIFAR-10, we apply a 13-layer convolutional neural network (ConvNet) to build our framework. At the first stage of unsupervised learning, 2n is generated via convolution and up-sampling layers. Similar as the first stage, 26 is generated via the encoder framework which is composed of convolution and dense layers. We maintain a moving average of the learner and use the difference between the predictions of learner and guider model to calculate e. The decoder in the semi-supervised learning stage is built with de-convolution and dense layers.
Due to the limitation in space, we detail the model architecture in the Appendix section. We use the Adam optimizer (Kingma & Ba 2014) with an initial learning rate of 0.001, 31 = 0.90 and 82 = 0.99. Each experiment has 5 runs and each run has 350 epochs. In each epoch, UMN uses all the training data including labeled and unlaheled data. For these supervised learning frameworks, we only use the labeled data. Our implementation is based on deep learning framework TensorFlow with a single NVIDIA Tesla P100 GPU.
4.2 DATASETS AND RESULTS
4.2.1 MNIST
MNIST is widely used dataset in machine learning community. It includes 60, 000 images with size 28 28 pixels. In our experiments, we use 50, 000 images for training and 10, 000 images for testing. 100 labeled data samples used in the experiments. In semi-supervised training, each mini-batch have 5 labeled samples and 95 unlabeled examples. In our compared supervised model, we apply a Multi-Layer Perceptron (MLP) classifier with one hidden layer of 784 units with ReLU activation. All the results are listed in T
From the results, we find that UMN performs much better than other approaches. Besides UMN, M-VAE (Langevin et al. 2018) outperforms other benchmarks. One possible reason may be due to probablistic modeling of the uncertainty. However, M-VAE depends on the pre-defined label corruption ratio which can't 't be obtained in most real scenarios.
We did not run Reweight (Ren al., 2018) on the MNIST dataset since at the time of running the experiments an implementation of the approach on noisy MNIST data was not available and our implementation of the approach would not be a fair comparison without having the right parameter settings. However, comparisons are shown on the other datasets as implementation for them was made available by the authors.
4.2.2 SVHN
We experiment with the Street View House Numbers (SVHN) datasets. This dataset includes 73, 257 RGB images of 32 x 32 resolution belonging to ten different classes. Following the same experimental settings as in (Tarvainen data samples per category and we use the rest of the images for unlabeled data in the semi-supervised learning setting. We randomly corrupt the sample labels within each category uniformly with our defined corruption ratio. We also compare with other five different models on SVHN dataset. In our experiment, each batch includes 5% labeled data.
Valpola| 2017), we use 500 labeled data samples from SVHN where 50
The comparison results are given in Table [ As listed in this table, we can find that UMN behaves the best in general except for the case when the corruption ratio is 10%. However, we do not observe this phenomena in MNIST and CIFAR-10. One of the reasons may be that MT can handle the situation when the noisy data occupy a small portion. The performance of MT model drops significantly as ingest more mislabeled data. From the result, we also observe that other two robust learning
7

---

Under review as a conference paper at ICLR 2020
Table 1: Comparison of results on different benchmarks. S-1 represents the supervised learning model via 13 conv layers and S-2 represents supervised learning model via ResNet-101. MN and RW indicate the MentorNet and Reweight approach respectively. Error rate percentage % is used as the measurement unit.
Approaches
Dataset
Corruption Ratio
S-1
S-2
MT
M-VAE
MN
RW
UMN(ours)
10%
29.4
29.5
6.6
4.3
25.6
2.5 + 0.14
20%
36.2
36.6
11.7
2.7
39.7
2.8 0.37
MNIST
30%
41.1
43.0
14.6
45.8
40%
51.3
49.0
17.9
9.9
57.7
5.8 3.8
50%
57.5
59.4
34.4
28.0
65.3
18.0 + 12.0
10%
34.9
32.0
18.0
26.1
29.1
27.3
23.9 0.2
20%
46.3
46.0
45.5
35.2
42.1
39.0
30.5 0.3
SVHN
30%
61.4
58.9
53.0
37.2
53.8
51.9
0.6
40%
61.8
60.5
63.9
40.1
59.1
53.7
37.1
50%
65.1
63.3
65.9
48.3
62.0
57.9
43.2 1.4
10%
43.2
41.3
39.2
41.2
41.2
39.4
37.8 + 0.5
20%
46.3
44.7
42.1
43.9
42.4
41.9
39.8 + 0.8
CIFAR- 10
30%
52.9
50.0
47.8
51.4
51.6
49.9
43.5 0.5
40%
57.1
56.9
52.2
52.1
54.1
53.9
43.5 1.2
50%
63.3
61.3
65.4
56.3
57.1
58.3
51.9 + 1.7
approaches (MentorNet and Reweight) achieve better performance than supervised only but much lower than these Semi-supervised learning methods. One possible reason is that they do not utilize the unlabeled data.
As illustrated in Fig- overfitting is observed in the M-VAE higher ratio of corruption. As we discussed previously, one major reason is because of the pre-defined f()) which is used as the constant value during the training phase. In contrast, UMN applies the sample-wise uncertainty estimation. It is accurate and more close to the real scenario. Furthermore, the training of UMN also converges faster with better performance.
Moreover, we conduct the experiments and evaluate the model performances using varying numbers of labeled data with the SVHN. We compare the performance for a popular supervised robust learning approach MentorNet(MIN) and UMN, for which the test errors shown in Table 2 In UMN, we still utilize all the unlabeled data which are not utilized for labeled data. UMN shows better performance for labeled data less than 1000 labels per class, which suggests the important role of the semi-supervised learning in robust learning and hence justify the purpose of this work.
Table 2: Test errors with Varying Number of Labels for SVHN
Number of labels
1000
2000
3000
10000
20000
UMN
23.9 0.2%
18.1%
16.3%
14.8%
13.2%
MentorNet
30.5 + 0.3%
27.9%
25.1%
18.3%
11.2%
4.2.3
CIFAR-10 10
Next, we run the comparisions on CIFAR-10 dataset (Krizhevsky 2009 This dataset contains 32 32 pixels RGB images belonging to ten different classes. To have fair comparison, we follow the same experimental setting as used in (Trvaainen & Valpola 2017). We randomly select 100 data samples from each category. In total, 4, 000 labeled data samples included in the training set. The rest of the training data are used for unlabeled data. The results are summarized in Table [I] From the listed results, can see that UMN achieves the best performance among all experiments.
8

---

Under review as a conference paper at ICLR 2020
reto
UNN lemigrt
Ernor rake
Ertor rite
Epocht
Epochs
Epachs
(a) 20% coruuption ratio
(b) 30% corruption ratio
(c) 40% corruption ratio
Figure 3: Illustration of eπror rates' comparisons between UMN and other benchmarks on SVHN during training stage under different corruption ratios (20%, 30% and 40%). UMN is less prone to overfitting on mislabeled data at higher corruption ratio.
4.3 IDENTIFYING SAMPLES WITH CORRUPT LABELS
We have conducted experiments using the MNIST-digit dataset to quantify how well UMN can identify samples whose labels are corrupted. We say, sample's label is corrupted if there is disagreement of the predictions between the Learner and the Guider models. That is, if the estimated >> 0 (Eq. I for given sample. Table Tablee3ssmm summarizes the accuracy identifying samples with corrupt labels. The sample-wist wise estimate of c estimated in the final epoch of training is used in the analysis. From the listed results, we can see that UMN has demonstrated very promising potential for filtering out noisy data. It could be easily adopted to other related works the data pre-processing step. Fig. 4b) shows the progression of the uncertainty estimate e while training on CIFAR- dataset with label corruption rate of 20%. We can see that the distribution is bi-modal and that the bi-modal separation increases as the training progresses. i.c., our approach becomes more confident in differentiating samples with corrupt labels from samples with correct labels.
In addition, to better understand the improved performance of UMN in these experiments. We compare the structure of high-dimensione latent feature representations obtained from two VAE components between the model with and the one without using label uncertainty estimation module. We apply t-SNE the latent feature vector (z,) on CIFARIO test dataset.
As shown in Fig. 4a is clear that the t-SNE maps exhibit different distributions between two architectures. In particular, as indicated by the feature distribution within the black and red boxes, UMN produces a much better separate map between different classes. The used training data had a label corruption rate of 30%.
In Fig. 5 we show the distribution of the embeddings for the training samples using different approaches. t-SNE is used for visualizing in 2-dimensions. The training data has 100 labeled samples with 50% label corruption. We can see that the supervised approach clearly overfits on the noisy labels. The semi-supervised approach used is better but one can still see that some of the classes are confused with each other due to label noise, especially when looking at the central region of the plot. However, can be seen that using the proposed UMN approach results in better intra-class grouping and inter-class separation of training data
Table 3: Performance of identifying samples with corrupt labels.
corruption ratio
10%
20%
30%
40%
50%
Recall
1.0
0.85
0.71
0.85
0.8
Precision
0.32
0.55
0.71
0.79
0.7
Fl score
0.48
0.67
0.71
0.82
0.75

---

Under review as a conference paper at ICLR 2020
Number of labeled dataa
Without UMN
150
100
0.00
Epoch
0.33
50
Epsilon
0.66
With UMN
(a) 2D latent fcature visualization
(b) Progression of the estimated
Figure 4: Illustration of latent feature distribution and progression of (a) 2D embedding of the learned latent feature 2n (output of encoder in the VAE which serves as the input to the classifier branch) obtained using t-SNE (Maaten & Hinton 2008 for the MNIST test data. Left and Right plots show distributions obtained without and with the use of uncertainty estimate respectively. Different colors the plots represent different classes (total 10). (b) The distribution visualization of evolves along with the time.
(a) Supervised approach
(b) supervupe approach
(c) UMN
Figure 5: Plots showing the distribution of MNIST training samples in the learned latent space for different approaches. We have used I-SNE to visualize the distribution in 2-dimensions. The different colors indicate the 10 different classes of MNIST. "Circles" used to represent true labels and "Cross" to represent mislabeled samples. We can see that similarly colored circles (same class) are better grouped by proposed UMN approach when compared supervised and semi- semi-supervised approaches. For the experiment, we use the MNIST dataset with 100 labeled samples with 50% label corruption. The network architecture used for UMN is described in section 4.2. 1 Using the same architecture, for the supervised experiment we train only the classifier branch the model, and for the semi-supervised experiment we the uncertainty estimate to be
10

---

Under review as a conference paper at ICLR 2020
5 CONCLUSION
In this work, we target a new problem of learning on partially labeled data with noise and propose novel framework (UMN). UMN is semi-supervised deep generative model dealing with data including scarce and potentially corrupted labels. Moreover, UMN can explicitly estimate the label uncertainty for given potentially mislabeled data sample. Furthermore, we integrate these frame- works into an end-to-end pipeline. Compared to previous works, we do not need the subset of clean label data or any preknowledge of the corruption ratio. UMN is able to estimate the label correctness based on the uncertainty calculation. Experimental results demonstrate the superiority of UMN over state-of tfftthr-rrtss for training deep learning models with noisy labels.
REFERENCES
Yifan Ding, Liqiang Wang, Deliang Fan, and Boging Gong. A semi-supervised two-stage approach to learning from noisy labels. In WACV. IEEE, 2018.
Jacob Goldberger and Ehud Ben-Reuven. Training deep neural-networks using noise adaptation layer. 2016.
Jonathan Gordon and José Miguel Hemmández-Lobato, Bayesian semisupervised learning with deep generative models. arXiv preprint arXiv:1706.0975|, 2017.
Ryuichiro Hataya and Hideki Nakayama. Unifying semi-supervised and robust learning by mixup. In ICLR The 2nd Learning from Limited Labeled Data (LLD) Workshop, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image 1at.bb//1122333888
recognition. CoRR, abs/1512.03332 2015.
Stanislaw Jastrzebski. Zachary Kenton, Devansh Arpit, Nicolas Ballas, Asja Fischer, Yoshua Bengio, and Amos J. Storkey. Three factors influencing minima in SGD. CoRR, abs/1711.04623, 2017.
URL http::ttpp/:iiiie org/ abs/1711 04623
Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, and Li Fei-Fei. Mentornet: Learning data-driven curriculum for very deep neural networks on corrupted labels. arXiv preprint 1rXiv:1712.05055, 2017.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 141114412....... 2014.
Diederik P. Kingma. Danilo Jimenez Rezende, Shakir Mohamed, and Max Welling. Semi-supervised learning with deep generative models. CoRR, abs/1406.55 52298, 2014. URL http:://axxiv
org/ Torg/abs/440 1006./1406.299
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
Maxime Langevin, Edouard Mchlman, Jeffrey Regier, Romain Lopez. Michael I. Jordan, and Nir Yosef. A deep generative model for semi-supervised classification with noisy labels. CoRR, abs/1 00//0089955577 2018. httl:/c:pp///ax....otr/////////
05957
Yuncheng Li, Jianchao Yang. Yale Song, Liangliang Cao, Jiebo Luo, and Li-Jia Li. Learning from noisy labels with distillation. In ICCV, 2017.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne, Journal of machine learning research, (NNov):2579-266)) 2008.
Stephan Mandt, Matthew D. Hoffman, and David M. Blei. Stochastic gradient descent as approximate bayesian inference. Journal of Machine Learning Research,
2017. URL
http::/ http://....//ppp///////11111 htm
Aditya Menon, Brendan Van Rooyen, Cheng Soon Ong, and Bob Williamson. Learning from corrupted binary labels via class- paosabrobaaiility estimation. In Francis Bach and David Blei (eds.). ICML, Lille, France, 07-09 Jul 2015. URL hatt:://proceedng mlr. wwwss/vessv v
menonn 5. htmll
11

---

Under review as a conference paper at ICLR 2020
Nagarajan Natarajan, Inderjit S. Dhillon, Pradeep Ravikumar, and Ambuj Tewari. Learn ing with noisy labels. In NIPS, 2013. URL htttp:/papers wwwp.cc/papp.......ccccepppppppe 507
Giorgio Patrini, Alessandro Rozza, Aditya Krishna Menon, Richard Nock, and Lizhen Qu. Making deep neural networks robust to label noise: A loss correction approach. In CVPR, July 2017.
B. T. Polyak and B. Juditsky. Acceleration of stochastic approximation by averaging. SIAM J. Control Optim., 30(4):838 855, July 1992. ISSN 0363-0129. doi: 10.1137/0330046, URL http://dkx doi.
http:///w......r///0.1111133/332 1ttp://1..117703300000
Scott E. Reed, Honglak Lee, Dragomir Anguelov, Christian Szegedy, Dumitru Erhan, and Andrew Rabinovich. Training deep neural networks noisy labels with bootstrapping. In ICLR, 2015.
URL wttpbhttp///aaxl..orrg.g...
6596)
Mengye Ren, Wenyuan Zeng, Bin Yang, and Raquel Urtasun. Learning to reweight examples for robust deep learning. In ICML, 2018.
Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. In NIPS. Curran Associates, Inc., 2017.
APPENDIX
6.1 ELBO FOR VAE UNCERTAINTY MININING NET (UMN)
We start from the Jensen's inequality for the log probability of the observations,
log P(X) 2 By(z/a)) log P(X - KL(g(2|X)) || p(z))
(14)
where the observed variables X {0.X) } and the latent variables for the VAE structure proposed with a stacked semi-supervised learning architecture. The posterior and likelihood factorizes
( 2b. v|., 0) q(2a (r) g(vl (<oole.1 ((2lk)) v) p(r, ú 3م 2b,4) p(x[==)) p()l,,
The reconstruction loss on r.h.s of Eq. 14 then gives,
Σ
[logP(x(20)) logl(0)... v)]
SEC
Eps(aal log P(r)za)) + Eys(tsla) (n)
Σ
Lo (uk za zog Po Po Za. M1c
22 BCEC
Ega(tale) log P(x|=a) + Bys (e)
Σ
(vn (2.) log Pa za),
(15)
BLEO
where 4s and pe are the encoder and decoder respectively for the variational autoencoder and can be parametrized by deep neural network. We note that the second term in the last equality of Eq. 15 is summed over the labeled samples in Semi-supervised learning. The KI. divergence in Eq. 14 given
12

---

Under review as a conference paper at ICLR 2020
(L(a(Zas 2b. 8/X, 2) p(za, 2b. v))
po(yl [o) Boa(zala) E Egalzs sa.1)
log (0 + log (vl Ra) + log (0(22) y)
MEC
-logp(zal<>. v) logg(zs) - log pogp(v)
Σ 2o (KL()) v) p(2))))
NEC
- -Ego(z.la))
Σ q (2) (KL p(v)))
NEC
-Eoo (a)
Σ
4 (v Za E.
go (ool=
log Po v) - log (<(=ul=))
(16)
NЕC
To summarize, the ELBO for VAE of UMN
B (2(a(2 (2)
Σ (KI p(=)))
(17)
(vl (KL( p(V)))
(xys (m)
Σ
(y Za (o (log loggo(zzllz0 2b. v) log ao (zal))))
+ (2al(z))) log P(z(za)) (xo (x)
Σ go (2a) log Po (u ( zn),
(18)
ILEC
where in the last term, estimations for probabilities of the observed label given based on the true labels and the corresponding encoded sample Za. The loss given by the ELBO are summed over all the samples except for the last term summed over the samples with the observed variable available.
6.2 ALGORITHM OF UMN
Algorithm 1: UMN Algorithm
while Traning() do
Sample minibatch from dataset 5
Draw = from qs (x0))x))) given of St: Draw 2 from go (=)]=²);
for all data x² E S:: do
I y1 ~ (=(())2)) end
for all labeled data 2², E S": do
ÿi ~ pe pel y, 2²) Compute based on the uncertainty estimation Eq.9) via Learner and Guider
end
Compute the generative process via Eq. 30 Compute loss function via Eq.
8C.
#C σΦ
00 Je
0= = 0 AdamUpdate(O);;
AdamUpdate(d)):
For ó in classifier y; o% (v, [x)) maintain Polyak's moving average for the guider (t) at 1) + a ot())
end
13

---

Under review as a conference paper at ICLR 2020
6.3 MODEL ARCHITECTURE OF UMN WITH CONVOLUTIONAL NEURAL NETWORK
The model architecture is listed in the table 4 The plot describe how the classifier and VAE model is setup. The VAE a stacked autoencoder with M1 + M2 Kingma al. (2014). M1 is variational autoencoder with convolution neural networks as the encoder and decoder. M2 is composed of only fully connected layers.
Table 4: Configuration of the neural networks
Layers of Classifier Input
Hyperparameters
32 x 32 RGB
Translation (shared with MVAB)
Randomly [ōx; dy] [-2,
Gaussian noise (shared with MVAE)
a= 0.15
Conv2D (shared with MVAE)
filter size: (3, 3, 128), same padding
Conv2l) (shared with MVAE)
filter size: (3, 3, 128), same padding
Conv2D (shared with MVAE)
filter size: (3, 3, 128), same padding
Pooling (shared with MVAB)
MaxPool, (2, 2)
(shaout(shared with MVAE)
p= 0.5
Conv2D (shared with MVAE)
filter size: (3, 3, 128), same padding
Conv2D (shared with MVAE)
filter size: (3, 3, 128), same padding
Conv2D (shared with MVAE)
filter size: (3, 3, 128), same padding
Pooling (shared with MVAE)
MaxPool, (2, 2)
Dropout (shared with MVAE)
p= 0.5
Conv2D (shared with MVAE)
filter size: (3, 512), valid padding
Conv2D
filter size: (1, 256), valid padding
Conv2D
filter size: (1, 1, 128), valid padding
Pooling
AvgPool, (6,6)
Fully Connected + Softmax
128 + 10
Layers of UMN
Hyperparameters
Conv2D
filter size: (3, 3, 32), same padding
Flatten
(8,8,32) 1152
Fully Connected
1152 + 256
Fully Connected
256 + 1152
Reshape
1152 - (6, 6, 6,32)
Conv2D Transpose
66,6,32) (8, 128)
ip-sampling
Conv2D
filter size: (3, 3, 128), same padding
Conv2D
filter size: (3, 3, 128), same padding
Conv2D
filter size: (3, 3, 128), same padding
Up-sampling
Conv2D (shared with MVAE)
filter size: (3, 3, 128), same padding
Conv2D (shared with MVAE)
filter size: (3, 3, 128), same padding
Conv2D (shared with MVAE)
filter size: (3, 3, 128), same padding
6.4 UNCERTAINTY ESTIMATION AS DIFFERENCE BETWEEN THE GUIDER AND THE LEARNER
We illustrate this using with a model near the convergence in a convex domain Mandt et al. 2017). assuming a small label corruption ratio. This suggests that our calculated uncertainty can help focus more on the correct labeled samples.
Assumption We assume that in the optimization problem, the stationary distribution is constrained to convex region, where the loss function has quadratic form
1 L= 0 OTH H0,
(19)
2
where H. is the Hessian of the loss surface near minimum and the vector 0 of the model parameters. Without loss of generality, optimal lies on the origin.
14

---

Under review as a conference paper at ICLR 2020
classifier
AVG fealing 14.1
e1 1,500
A. 1 1,000
MVAE
2DX (2A.128)
022 11/1900
1531111111 (21)
(22)
194)
Food 2elg. 21
liporpiig
122. 28,
06.6,
18.0 120
(1152)
Corn 20
D6, (6,199)
Upparpllzz
390
110. 04
Carv 2016 33,30)
192,52,115
Rocossiuteelise 532. St,
(39,30.1
Figure 6: The network structure for Variational Autoencoder for UMN used for SVHN and CIFARIO. The network has two modules. The first one is classifier composed of convolution, pooling, dense and softmax layers. The second one is M1 + M2 variational autoencoder. The 21 of M1 + M2 and the classifier share the same encoder. 21 and x2 are the two variational autoencoder that encode and decode the raw image.
6.4.1 THE CONVERGENCE BEHAVIOR AS STOCHASTIC PROCESS
The convergence hehavior of Stochastic Gradient Descent (SGD) can be described by a stochastic differential equation
db -tg(O)dt À B(0)dW(1), VS
(20)
where g0) = HO(()) is the gradient for the weights and S is the mini-batch size. d(W( (t) represents Wienner process stochastic gradient descent process. The covariance of the SGD process Σ EE(0T) satisfies HE ΣH A BB X and É represent the step size and step index respectively.
stochastic gradient descent. (0) is introduced due to the noise in
Lemma 2. The stochastic process for parameter optimization of deep learning problem using only piecewise linear activation function with and without noisy labels can be described by the following Wienner processes,
À B(0)d((t);
dl(t) XALO(t)dt +
(21)
VS
dl(t) -HHO(t)dt+ + VS B(O)
(22)
where H. = H. and B = B, assuming the learning rate and batch size is the same.
The proof follows directly from theorem (Theorem 4) from (Patrini et al. 2017). stating that the curvature the loss surface (Hessian) is invariant with respect to the noise when the neural network only uses piecewise linear function its activation function.
15

---

Under review as a conference paper at ICLR 2020
6.4.2 POLYAK'S AVERAGE FOR THE WEIGHTS OPTIMIZATION
Polyak al PPolyak & Juditsky 1992) proved that the iterate average gives the optimal convergence rate and approximates the optimal by using the average of the iterates online as
O 1) (l(+) (0)(
6 14+ + 1 Me
t 0t
(23)
and the iterate average after 7 step is
Pc(T)= PT /
0 (t)dt,
(24)
where 0r. and Og indicate the weights of the leaner and guider respectively. Next we prove theorem th in the presence of the noisy labels, when we update the weights of the learner model, the distance between the learner and the guider suggests the correctness of the labels assuming the invariance of the Hessian of the learner and guider by the lemma.
Proof.
(Vo- [10r- - -01l)) T Vo- [10,, - dall]
=E 07000 - - 1 T fg + T.
-[0]02] 1e000g rr T 0- + 00
00
kc
(25)
In order show Eq.
13
is equivalent show
E E 0z]
B00
(26)
We denote
0, = 0s + 6":
0g = 0r 0
These are two stochastic optimization process described above for the stochastic gradient descent and the iterate average. 0g and 0r can be seen as the deviation from the optimal point. Hence need to show
0 < 0r]
(27)
The left hand side of Eq. gives
E Tr(E)
(28)
In order to evaluate the right hand side of the equation, recall the Green's function for the Ornstein-Uhlenbeck process,
B(O(t)oT(s) (s)) Ee AP((0-0) if - if
t <
(29)
t
Then, the right hand side of Eq.
27 becomes,
E
0
0r
Tr
E (t)0g
T
Tr
e XH((t E2)
Tr
UU AUA
~ Tr
H - 1
(30)
XT
16

---

Under review as a conference paper at ICLR 2020
where assume that the Hessian is full rank and its inverse has eigen decomposition
=- UA-
(31)
When
1 1 (T(
T < A Tr {
(32)
The Eq. 13 holds.
If we make assumption as in Jastrzebski et al. (2017), i.e., covariance of the local minimal is approximated by the Hessian, namely, H Σ. We observe that the last equation in the proof above becomes
T> (T A Tr
(33)
XL
where C is the Laplacian. This indicates that our approach favors the loss surface around minimal of a lower mean curvature and hence narrower minimal.
17


Reviews:
[{"review_id": "SyeQxtH3tr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper describes a method for learning in semi-supervised settings with label noise. This is an interesting topic with a relatively scarce literature. The proposed method works by first postulating a generative model for the labelled/unlabeled/label-corrupted data. The model is then fitted using a standard variational lower bound maximization.\n\nThe approach is elegant, and appears to work empirically well. Unfortunately, I do not seem to really understand the rationale behind the main novelty (even at an heuristic level) of the paper which is contained in Section 3.2.1, namely Equation (10) which describes the intensity of the label corruption. Why is it a sensible idea? There is only 3 lines of comments after equation 10, which seems a bit short since this is the crux of the paper. Also, the notations of equation (10) are very confusing (to me) since the function f(.) was used before to denote another quantity.\nMinor remarks:\n=============\n\n1. page4 was a bit difficult to digest at first reading since the author did not describe in the main text he form (i.e. factorization structure) of the variational distribution q.\n\n2. second line of Equation 14 seems to be wrong, while the 1st and last line seems correct -- the authors may want to double check\n\n3. the term \"labeled loss term\" was not properly defined, which makes the reading a bit difficult even if one eventually gets what the authors mean\n\n4. In equation (6) I do not understand why f(eps) = log[(C-1)(1-eps)/eps]. Why isnt it equal to log[p(hat(y)|y)] ?\n\n5. notations z_a, z_1, etc.. do not seem to be consistent throughout the text\n\n6. it is quite surprising to me that, in the low data-corruption regime, the method appears to be competitive/better with consistency-based method such as the MT approach.\n\n7. naive question: why directly modeling \\eps as a function of z_a or (z_a, y), possibly parametrized by a neural network, a bad idea?\n\nIn conclusion, it feels like the method has a lot of potential (in views of the numerical simulations) -- if the authors could clarify the exposition, this could be a very good contribution to the field. The method appears to be conceptually simple (i.e. postulate a generative model + fit it by maximizing the ELBO + one trick to estimate \\eps), which is a good thing -- what is missing, I think, is a real discussion of why the proposed manner to estimate \\epsilon is sensible.\n\nEdit after reading [1] \n======================\nThe proposed generative model is same -- the authors should make this very clear in the paper. Although is is acknowledged: \"In this work, we implement the idea in (Langevin et al., 2018)\" -- this is only in section \"4.1 IMPLEMENTATION DETAILS\". After reading (1), it is clear that the novelty of the paper us much less than what I had initially thought. The only difference is in Section 3.2.1, and this Section is far from being satisfying.\n\n\n[1] \"A Deep Generative Model for Semi-Supervised Classification with Noisy Labels\", Langevin & al", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "Hye15ApRYr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors proposed a novel framework, Uncertainty Mining Net (UMN), to address the problem of learning on limited labeled data with label noise. First, UMN applied the unsupervised variational autoencoder to learn a more representative latent feature representation by involving large amounts of unlabeled data. Second, in the training process, ELBO integrates information from all the training data and sample-wise uncertainty estimated from the predictions of the updating model and its exponential moving average is incorporated to enable the learning process to focus on the data with reliable labels. Experimental results show that UMN outpuroms several state-of-the-art methods on multiple benchmark datasets. \n\nUMN will be helpful for a lot of real-world products since annotation is expensive and annotation quality is a regular concern. However, it is subjectively uncertain to define whether it is a large or small dataset without considering application context and model complexity. It would be better if the authors can further provide the error rate changes under different number of labeled training data. Such analysis would provide suggestions regarding when it is necessary to implement UMN. On the other hand, it would be nice to explore how tolerant/sensitive the UMN is to the corruption rate to every single class, especially for a multi-classification problem. In the real world, the data within the same class might be labeled incorrectly more than correctly. It would be better to investigate if UMN is able to identify such situation and correct the labels accordingly. ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "Hkx78mZ3tS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Update after rebuttal:\nThe rebuttal addressed a few of my concerns, but there is still a major issue. Namely, UMN is claimed to work on sample-wise label noise, but there are no experiments to support this (note that this is different from non-uniform class-dependent label noise). As fixing this would require large modifications to the paper, I am keeping my score at weak reject.\n\n----------------------------------------------------------------------------------\nSummary:\nThis paper presents a method for training classifiers in the setting of semi-supervised learning with noisy labels. The proposed method, Uncertainty Mining Net, combines the Mean Teacher method of Tarvainen and Valpola with the M-VAE method of Langevin et al. to estimate the trustworthiness of each input-label pair and weigh its contribution to the loss. This is not an obvious use of the Mean Teacher method, and it seems like a nice idea.\n\nThe results are good when controlling for architecture, and the setting is important and underexplored, but there are several concerns I have with the paper in its current form and some parts I would like clarified. At present, the paper is borderline, and I will raise my score if these are addressed.\n\nMajor points:\nThe generative process defined in equations 2 and 3 presents a model for input-dependent label noise, but the corruptions in the experiments are conditionally independent of the input given the true label. What is p(y_tilde | y, z_a) supposed to capture when the true noise model in the experiments follows p(y_tilde | y)?\n\nIt seems like the proposed approach would have difficulty with non-uniform label noise, but there is no discussion on this. Adding discussion of this would be good.\n\nWhat is the purpose of z_b? It seems like a redundant variable in the generative process, since it is only used with y to sample z_a, and it seems like z_a could just be sampled from y.\n\nThe caption in Figure 1 says, “For simplicity, we omit the optional consistency loss term between the classifiers of θ and θ 0 for unlabeled data in the figure”, but this is never mentioned again. I would find it interesting to see the combined effect of the Mean Teacher consistency loss with the VAE reconstruction loss, since they are distinct approaches to semi-supervised learning. Did you run experiments with the consistency loss?\n\nMinor points:\nThe writing is full of grammatical errors and typos, including\n\n“we experiment a CNN architecture with 13 convolutional neural network as well as a ResNet-101 architecture”\n\n“the training of UMN also converge faster”\n\n“For the details of the proof, please refer to the appendix section.”\n(This quote is from the appendix!)\n\n“eign decomposition”\n\n“for solving learning model”", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents an interesting idea but all reviewers pointed out problems with the writing (eg clarity of the motivation) and with the motivation of the experiments and link to the contest. The rebuttal helped, but it is clear that the paper requires more work before being acceptable to ICLR.
Decision: Reject