Title: Generative Noisy-Label Learning by Implicit Dicriminative Approximation with Partial Label Prior
Venue: NeurIPS 2023
Accepted: no
Keywords: noisy label learning; partial label learning; implicit generative model
Abstract: The learning with noisy labels has been addressed with both discriminative and generative models. Although discriminative models have dominated the field due to their simpler modeling and more efficient computational training processes, generative models offer a more effective means of disentangling clean and noisy labels and improving the estimation of the label transition matrix. However, generative approaches maximize the joint likelihood of noisy labels and data using a  complex formulation that only indirectly optimizes the model of interest associating data and clean labels. Additionally, these approaches rely on generative models that are challenging to train and tend to use uninformative clean label priors. In this paper, we propose a new generative noisy-label learning approach that addresses these three issues. First, we propose a new model optimisation that directly associates data and clean labels. Second, the generative model is implicitly estimated using a discriminative model, eliminating the inefficient training of a generative model. Third, we propose a new informative label prior inspired by partial label learning as supervision signal for noisy label learning. Extensive experiments on several noisy-label benchmarks demonstrate that our generative model provides state-of-the-art results while maintaining a similar computational complexity as discriminative models. *Code will be available if paper is accepted*.

Abstract

The learning with noisy labels has been addressed with both discriminative and generative models. Although discriminative models have dominated the field due to their simpler modeling and more efficient computational training processes, generative models offer a more effective means of disentangling clean and noisy labels and improving the estimation of the label transition matrix. However, generative approaches maximize the joint likelihood of noisy labels and data using a complex formulation that only indirectly optimizes the model of interest associating data and clean labels. Additionally, these approaches rely on generative models that are challenging to train and tend to use uninformative clean label priors. In this paper, we propose a new generative noisy-label learning approach that addresses these three issues. First, we propose a new model optimisation that directly associates data and clean labels. Second, the generative model is implicitly estimated using a discriminative model, eliminating the inefficient training of a generative model. Third, we propose a new informative label prior inspired by partial label learning as supervision signal for noisy label learning. Extensive experiments on several noisy-label benchmarks demonstrate that our generative model provides state-of-the-art results while maintaining a similar computational complexity as discriminative models. Code will be available once paper is accepted.

1 Introduction

Deep neural network (DNN) has achieved remarkable success in computer vision [13, 21], natural language processing (NLP) [10, 51] and medical image analysis [24, 38]. However, DNNs often require massive amount of high-quality annotated data for supervised training [9], which is challenging and expensive to acquire. To alleviate such problem, some datasets have been annotated via crowdsourcing [46], from search engines [35], or with NLP from radiology reports [38]. Although these cheaper annotation processes enable the construction of large-scale datasets, they also introduce noisy labels for model training, resulting in performance degradation. Therefore, novel learning algorithms are required to robustly train DNN models when training sets contain noisy labels.

The main challenge in noisy-label learning is that we only observe the data, represented by random variable X , and respective noisy label, denoted by variable ˜ Y , but we want to estimate the model p ( Y | X ) , where Y is the hidden clean label variable. Most methods proposed in the field resort to two discriminative learning strategies: sample selection and noise transition matrix. Sample selection [1, 12, 22] optimises the model p θ ( Y | X ) , parameterised by θ , with maximum likelihood optimisation restricted to pseudo-clean training samples, as follows

34

35

36

37

38

39

40

41

42

43

44

45

46

47

48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

80

81

and P ( X, ˜ Y ) is the distribution used to generate the noisy-label and data points for the training set. Note that E P ( X, ˜ Y ) [ clean ( X, ˜ Y ) × p θ ( ˜ Y | X ) ] ≡ E P ( X,Y ) [ p θ ( Y | X )] if the function clean ( . ) successfully selects the clean-label training samples. Unfortunately, clean ( . ) usually relies on the small-loss hypothesis [2] for selecting R % of the smallest loss training samples, which offers little guarantees of successfully selecting clean-label samples. Approaches based on noise transition matrix [44, 6, 32] aim to estimate a clean-label classifier and a label transition, as follows:

where θ = [ θ 1 , θ 2 ] , p θ 1 ( ˜ Y | Y, X ) represents a label-transition matrix, often simplified to be classindependent with p θ 1 ( ˜ Y | Y ) = p θ 1 ( ˜ Y | Y, X ) . Since we do not have access to the label transition matrix, we need to estimate it from the noisy-label training set, which is challenging because of identifiability issues [27], making necessary the use of anchor point [32] and regularisations [6].

On the other hand, generative learning models [3, 11, 50] assume a generative process for X and Y , as described in Fig. 1. These methods are trained to maximise the data likelihood p ( ˜ Y , X ) = ∫ Y,Z p ( X | Y, Z ) p ( ˜ Y | Y, X ) p ( Y ) p ( Z ) dY dZ , where Z denotes a latent variable representing a lowdimensional representation of the image, and Y is the latent clean label. This optimisation requires a variational distribution q ϕ ( Y, Z | X ) to maximise the evidence lower bound (ELBO): with

where p θ 1 ( X | Y, Z ) denotes an image generative model, p θ 2 ( ˜ Y | X,Y ) represents the label transition model, p ( Z ) is the latent image representation prior (commonly assumed to a standard normal distribution), and p ( Y ) is the clean label prior (usually assumed to be a non-informative prior based on a uniform distribution). Such generative strategy is sensible because it disentangles the true and noisy labels and improves the estimation of the label transition model [50]. A limitation of the generative strategy is that it optimises p ( ˜ Y , X ) instead of directly optimising p ( X | Y ) or p ( Y | X ) . Also, compared with the discriminative strategy, the generative approach requires the generative model p θ 1 ( X | Y, Z ) that is challenging to train. This motivates us to ask the following question: Can we directly optimise the generative goal p ( X | Y ) , with a similar computational cost as the discriminative strategy and accounting for an informative prior for the latent clean label Y ? In this paper, we propose a new generative noisy-label learning method to directly optimise p ( X | Y ) by maximising E q ( Y | X ) [log p ( X | Y )] using a variational posterior distribution q ( Y | X ) . This objective function is decomposed into three terms: a label-transition model E q ( y | x ) [log p (˜ y | x , y )] , an image generative model E q ( y | x ) [ log p ( x | y ) p ( y ) q ( y | x ) ] , and a Kullback-Leibler (KL) divergence regularisation term. We implicitly estimate the image generative term with the discriminative model q ( Y | X ) , bypassing the need to train a generative model. Moreover, our formulation allows the introduction of an instance-wise informative prior p ( Y ) inspired by partial-label learning [36]. This prior is re-estimated at each training epoch to cover a small number of label candidates if the model is certain about the training label. Conversely, when the model is uncertain about the training label, then the label prior will cover a large number of label candidates, which also serve as a regularisation of noisy label training. Our formulation only requires a discriminative model and a label transition model, making it computationally less expensive than other generative approaches [3, 11, 50]. Overall, our contributions can be summarized as follows:

We introduce a new generative framework to handle noisy-label learning by directly optimising p ( X | Y ) .

Our generative model is implicitly estimated with a discriminative model, making it computationally more efficient than previous generative approaches [3, 11, 50].

Our framework allows us to place an informative instance-wise prior p ( Y ) for latent clean label Y . Inspired by partial label learning [36], p ( Y ) is constructed for maintaining high coverage for latent clean label and regularise uncertain sample training.

We conduct extensive experiments on both synthetic and real-world noisy-label benchmarks that show that our method provides state-of-the-art (SOTA) results and enjoy a similar computational complexity as discriminative approaches.

2 Related Work 82

Sample selection. The discriminative learning strategy based on sample selection from (1) needs 83 to handle two problems: 1) the definition of clean ( . ) , and 2) what to do with the samples classified 84 as noisy. Most definitions of clean ( . ) resort to classify small-loss samples [2] as pseudo-clean [1, 85 4, 12, 15, 22, 30, 34, 40]. Other approaches select clean samples based on the K nearest neighbor 86 classification in an intermediate deep learning feature spaces [31, 39], distance to the class-specific 87 eigenvector from the gram matrix eigen-decomposition using intermediate deep learning feature 88 spaces [17], uncertainty measures [19], or prediction consistency between teacher and student 89 models [16]. After sample classification, some methods will discard the noisy-label samples for 90 training [4, 15, 30, 34], while others use them for semi-supervised learning [22]. The main issue with 91 this strategy is that it does not try to disentangle the clean and noisy-label from the samples. 92

93

94

95

96

97

98

99

Label transition model. The discriminative learning strategy based on the label transition model from (2) depends on a reliable estimation of p ( ˜ Y | Y, X ) [6, 32, 44]. Forward-T [32] uses an additional classifier and anchor points from clean-label samples to learn a class-dependent transition matrix. Part-T [44] estimates an instance-dependent model. MEDITM [6] uses manifold regularization for estimating the label-transition matrix. In general, the estimation of this label transition matrix is under-constrained, leading to the identifiability problem [27], which is addressed with the formulation of anchor point [32], or additional regularisation [6].

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

Generative modelling. Generative modeling for noisy-label learning [3, 11, 50] explores different graphical models (see Fig. 1) to enable the estimation of clean labels per image. Specifically, CausalNL [50] and InstanceGM [11] assume that the latent clean label Y causes X , and the noisy label ˜ Y is generated from X and Y . Alternatively, NPC [3] assumes that X causes Y and proposes a post-processing calibration for noisy label learning. One drawback of generative modeling is that instead of directly optimising the models of interest p ( X | Y ) or p ( Y | X ) , it optimises the joint distribution of visible variables p ( X, ˜ Y ) . Even though maximising the likelihood of the visible data is sensible, it only produces the models of interest as a by-product of the process. Furthermore, these methods require the computationally complex training of a generative model, and usually rely on non-informative label priors.

Clean label prior. Our clean-label prior p ( Y ) constrains the clean label to a set of label candidates for a particular training sample. Such label candidates change aims to 1) increase clean label coverage, and 2) represent uncertainty of the prior. Increase coverage improve the chances of including latent clean label in supervision. For noisy samples, increase the number of candidates in p ( Y ) regularise noisy label training. Such dynamic prior distribution may resemble Mixup [53], label smoothing [28] or re-labeling [22] techniques that are commonly used in label noise learning. However, these approaches do not simultaneously follow the two design principles mentioned above. Mixup [53] and label smoothing [28] are effective approaches for designing soft labels for noisy label learning, but both aim to increase coverage, disregarding label uncertainty. Re-labeling switches the supervisory training signal to a more likely pseudo label, so it is very efficient, but it has limited coverage.

Partial label learning In partial label learning (PLL), each image is associated with a candidate label set defined as a partial label [36]. The goal of PLL is to predict the single true label associated with each training sample, assuming that the ground truth label is one of the labels in its candidate set. PICO [37] uses contrastive learning in an EM optimisation to address PLL. CAV [52] proposes class activation mapping to identify the true label within the candidate set. PRODEN [29] progressively identifies the true labels from a candidate set and updates the model parameter. The design of our informative clean label prior p ( Y ) is inspired from PLL, but unlike PLL, there is no guarantee that the multiple label candidates in our prior contain the true label. Furthermore, the size of our candidate label set is determined by the probability that the training sample label is clean, where a low probability induces a prior with a large number of candidates for regularising training.

136

137

138

139

140

141

142

143

144

145

146

3 Method

We denote the noisy training set as D = { ( x i , ˜ y i ) } |D| i =1 , where x i ∈ X ⊂ R H × W × C is the input image of size H × W with C colour channels, ˜ y i ∈ Y ⊂ { 0 , 1 } |Y| is the observed noisy label. We also have y as the unobserved clean label. We formulate our model with generative model that starts with the sampling of a label y ∼ p ( Y ) . This is followed by the clean-label conditioned generation of an image with x ∼ p ( X | Y = y ) , which are then used to produce the noisy label ˜ y ∼ p ( ˜ Y | Y = y , X = x ) (hereafter, we omit the variable names to simplify the notation). Below, in Sec. 3.1, we introduce our model and the optimisation goal. In Sec. 3.2 we describe how to construct informative prior, and the overall training algorithm is presented in Sec. 3.3.

3.1 Model

We aim to optimize the generative model log p ( x | y ) , which can be decomposed as follows:

In (4), p ( y ) represents the prior distribution of the latent clean label. The optimisation of p ( x | y ) can 147 be achieved by introducing a variational posterior distribution q ( y | x ) , with: 148

149

150

151

152

153

154

155

156

157

158

159

160

161

where KL [ . ] denotes the KL divergence, and

Based on Eq. (5) and (6), the expected log likelihood of p ( x | y ) is defined as

In Eq. (7), we parameterise q ( y | x ) and p (˜ y | x , y ) with neural networks, as depicted in Figure 2. The generative model p ( x | y ) usually requires to model infinite number of samples based on conditional label and a generative model is hard to capture such relationship. However, since noisy label learning is a discriminative task and classification performance is our primary goal, the generation can be approximated with with finite training samples, which is given training set. More specifically, we defines p ( x | y ) only on data points { x i } |D| i =1 by maximising -KL [ q ( y | x ) ∥ p ( x | y ) p ( y )] for a fixed q ( y | x ) , with the optimum achieved by:

Hence, the generative conditional p ( x | y ) can only represent the values of x within training set given the latent labels in y . This allow us transform discriminative model into implicit generative model without additional computation cost.

3.2 Informative prior based on partial label learning

In Eq. (7), the clean label prior p ( y ) is required. As mentioned in Sec. 2, we formulate p ( y ) inspired 162 from PLL [29, 37, 52]. However, it is worth noting that PLL has the partial label information available 163 from the training set, while we have to dynamically build it during training. Therefore, the clean label 164 prior p ( y ) for each training sample is designed so that the hidden clean label has a high probability 165 of being selected during most of the training. On one hand, we aim to have as many label candidates 166 as possible during the training to increase the chances that p ( y ) has a non-zero probability for the 167 latent clean label. On the other hand, including all labels as candidates is a trivial solution that does 168

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

not represent a meaningful clean label prior. These two seemingly contradictory goals target the 169 maximisation of label coverage and minimisation of label uncertainty, defined by: 170

where ✶ ( . ) is the indicator function. In (9), coverage increases by approximating p ( Y ) to a uniform 171 distribution, but uncertainty is minimised when the clean label y i is assigned maximum probability. In 172 general, training samples for which the model is certain about the clean label, should have p ( y i ) = 1 , 173 while training samples for which the model is uncertain about the clean label, should have p ( y i ) < 1 174 with other candidate labels with probability > 0 . Therefore, the clean label prior is defined by: 175

where Z is a normalisation factor to make ∑ |Y| j =1 p i ( j ) = 1 , ˜ y i is the noisy label in the training set, c i denotes the label to increase coverage, and u i represents the label to increase uncertainty, both defined below. Motivated by the early learning phenomenon [25], where clean labels tend to be fit earlier in the training than the noisy labels, we maximise coverage by sampling from a moving average of model prediction for each training sample x i at iteration t with:

where β ∈ [0 , 1] and ¯ y ( t ) is the softmax output from the model that predicts the clean label from the data input x i . For Eq. (11), C ( t ) i denotes the categorical distribution of the most likely labels for the i th training sample, which can be used to sample the one-hot label c i ∼ Cat ( C ( t ) i ) . The minimisation of uncertainty depends on our ability to detect clean-label and noisy-label samples. For clean samples, p ( y i ) should converge to a one-hot distribution, maintaining the label prior focused on few candidate labels. For noisy samples, p ( y i ) should be close to a uniform distribution to keep a large coverage of candidate labels. To compute the probability w i ∈ [0 , 1] that a sample contains clean label, we use the sample selection approaches based on the unsupervised classification of loss values [22]. Then the label u i is obtained by sampling from a uniform distribution of all possible labels proportionally to its probability of representing a noisy-label sample, with

where round ( |Y| × (1 -w i )) represents the number of samples to be drawn from the uniform distribution rounded up to the closest integer.

3.3 Training

We can now return to the optimisation of Eq. (7), where we define the neural networks g θ : X → 194 ∆ |Y|1 that outputs the categorical distribution for the clean label in the probability simplex space 195 ∆ |Y|1 given an image x ∈ X , and f ϕ : X × ∆ |Y|1 → ∆ |Y|1 that outputs the categorical 196 distribution for the noisy training label given an image and the clean label distribution from g θ ( . ) . 197 The first term in the right-hand side (RHS) in Eq. (7) is optimised with the cross-entropy loss: 198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

where { ˆ y i,j } K j =1 ∼ Cat ( g θ ( x i )) , with Cat ( . ) denoting a categorical distribution. The second term in the RHS in Eq. (7) uses the estimation of p ( x | y ) from Eq. (8) to optimise the KL divergence:

where p i = [ p i ( j = 1) , ..., p i ( j = |Y| )] ∈ ∆ |Y|1 is the clean label prior defined in Eq. (10), c i is a normalisation factor, and ⊙ is the element-wise multiplication. The last term in the RHS of Eq. (7) is the KL divergence between q ( y | x ) and p (˜ y | x , y ) p ( y ) , which represents the gap between E q ( y | x ) [log p ( x | y )] and E q ( y | x ) [ log p (˜ y , y , x ) q ( y | x ) ] . According to the expectation-maximisation (EM) derivation [8, 18], the smaller this gap, the better q ( y | x ) approximates the true posterior p ( y | x ) , so the loss function associated with this third term is:

Our final loss to minimise is

After training, a test image x is associated with a class with g θ ( x ) . An interesting point about this derivation is that the implicit approximation of p ( x | y ) enables the minimisation of the loss in (16) using regular stochastic gradient descent instead of a more computationally complex EM algorithm [33].

4 Experiments

We show experimental results on instance-dependent synthetic and real-world label noise benchmarks with datasets CIFAR10/100 [20]. We also test on three instance-dependent real-world label noise datasets, namely: Animal-10N [35], Red Mini-ImageNet [15], and Clothing1M [46].

4.1 Datasets

CIFAR10/100 [20] contain a training set with 50K images and a testing of 10K images of size 32 × 32 × 3, where CIFAR10 has 10 classes and CIFAR100 has 100 classes. We follow previous works [44] and synthetically generate instance-dependent noise (IDN) with rates in {0.2, 0.3, 0.4 ,0.5}. CIFAR10N/CIFAR100N is proposed by [43] to study real-world annotations for the original CIFAR10/100 images and we test our framework on {aggre, random1, random2, random3, worse} types of noise on CIFAR10N and {noisy} on CIFAR100N. Red Mini-ImageNet is a real-world dataset [15] containing 100 classes, each containing 600 images from ImageNet, where images are resized to 32 × 32 pixels from the original 84 × 84 to enable a fair comparison with other baselines [48]. Animal 10N [35] is a real-world dataset containing 10 animal species with five pairs of similar appearances (wolf and coyote, etc.). The training set size is 50K and testing size is 10K, where we follow the same set up as [5]. Clothing1M is a real-world dataset with 100K images and 14 classes. The labels are automatically generated from surrounding text with an estimated noise ratio of 38.5%. The dataset also contains clean samples for training and validation but we only use clean test for measuring model performance.

4.2 Practical considerations

We follow commonly used experiment setups for all benchmarks described in Sec. 4.1. 1 For the hyper-parameter setup, K in (13) is set to 1, and β in Eq. (11) is set to 0.9. For w in Eq. (12), we follow the commonly used Gaussian Mixture Model (GMM) unsupervised classification from [22]. For warmup epochs, w is randomly generated from a uniform distribution. Note that the approximation of the generative model from (8) is done within each batch, not the entire the dataset. Also, the minimisation of L PRI ( . ) can be done with the reversed KL using KL [ c i × g θ ( x i ) ∑ j g θ ( x j ) ⊙ p i ∥ ∥ ∥ g θ ( x i ) ] .

238

239

240

241

242

243

244

245

246

247

248

249

This reversed KL divergence also provides solutions where the model and implied posterior are close. In fact, the KL and reversed KL losses are equivalent when ∑ j g θ ( x j ) has a uniform distribution over the classes in Y and the prior p i is uniform in the negative labels. We tried the optimisation using both versions of the KL divergence (i.e., the one in (14) and the one above in this section), with the reversed one generally producing better results, as shown in the ablation study in Sec. 4.4. For all experiments in Sec. 4.3, we rely on the reversed KL loss. For the real-world datasets Animal-10N, Red Mini-ImageNet and Clothing1M we also test our model with the training and testing of an ensemble of two networks. Our code is implemented in Pytorch and experiments are performed on RTX 3090.

4.3 Experimental Results

Synthetic benchmarks. The experimental results of our method with IDN problems on CIFAR10/100 are shown in Tab.1 and Tab.2. Compared with the previous SOTA kMEDITM [6], on CIFAR10, we

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

270

271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

286

achieve competitive performance on low noise rates and up to 16% improvements for high noise rates. For CIFAR100, we consistently improve 2% to 4% in all noise rates. Compared with the previous SOTA generative model CausalNL [50], our improvement is significant for all noise rates. The superior performance of our method indicates that our implicit generative modelling and clean label prior construction is effective when learning with label noise.

Real-world benchmarks. In Tab.3, we show the performance of our method on the CIFAR10N/100N benchmark. Compared with other single-model baselines, our method achieves at least 1% improvement on all noise rates on CIFAR10N, and it has a competitive performance on CIFAR100N. The Red Mini-ImageNet results in Tab.4 (left) show that our method achieves SOTA results for all noise rates with 2% improvements using a single model and 6% improvements using the ensemble of two models. The improvement is substantial compared with previous SOTA FaMUS [48] and DivideMix [22]. In Tab.4(right), our single-model result on Animal-10N achieves 1% improvement with respect to the single-model SELFIE [35]. Considering our approach with an ensemble of two models, we achieve a 1% improvement over the SOTA Nested+Co-teaching [5]. Our ensemble-model result on Clothing1M in Tab.5 shows a competitive performance of 74.4%, which is 2% better than the previous SOTA generative model CausalNL [50].

4.4 Analysis

Ablation The ablation analysis of our method is shown in Tab.6 with the IDN problems on CIFAR10. First row ( L CE ) shows the results of the training with a cross-entropy loss using the training samples and labels in D . The second row ( L CE + L CE _ PRI + L KL ) shows the result of our method, replacing the KL divergence in L PRI as defined in (14), by a soft version of cross entropy loss. Next, the third row ( L CE + L PRI + L KL ) shows our method with the loss defined in (16). As mentioned in Sec. 4.2, these two forms provides similar solution where the model and implicit posterior are close and L PRI reverse generally performs better. In the fourth row ( L CE + L PRI reversed) by optimising the lower bound to E q ( y | x ) [log p ( x | y )] and finally the last row by optimising the whole objective function from (16) in the last row ( L CE + L PRI reversed + L KL (Ours)). In general, notice that the reversed L PRI improves the results; the KL divergence in L PRI works better than the CE loss; and the optimisation of the whole loss in (16) is better than optimising the lower bound, which justifies the inclusion of L KL ( . ) in the loss.

Coverage and uncertainty visualisation We visualise coverage and uncertainty from Eq. (9) at each training epoch for IDN CIFAR10/100 and CIFAR10N setups. In all cases, label coverage increases as training progresses, indicating that our prior tends to always cover the clean label. In fact, coverage reaches nearly 100% for CIFAR10 at 20% IDN and 97% for 50% IDN. Furthermore, for CIFAR100 at 50% IDN, we achieve 82% coverage, and for CIFAR10N "worse", we reach 92% coverage. In terms of uncertainty, we notice a steady reduction as training progresses for all problems, where the uncertainty values tend to be slightly higher for the problems with higher noise rates and more classes. For instance, uncertainty is between 2 and 3 for the for CIFAR10's IDN benchmarks, increasing to be

287

288

289

290

291

292

293

294

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

0

20

40

60

80

100

Epoch

0

20

40

60

80

100

Coverage

0

2

4

6

8

10

Uncertainty

0.2 Cov

0.5 Cov

0.2 Unc

0.5 Unc

(a) CIFAR10-IDN

0

20

40

60

80

100

Epoch

0

20

40

60

80

100

Coverage

0

20

40

60

80

100

Uncertainty

0.2 Cov

0.5 Cov

0.2 Unc

0.5 Unc

(b) CIFAR100-IDN

0

10

20

30

40

50

60

70

80

Epoch

0

20

40

60

80

100

Coverage

0

2

4

6

8

10

Uncertainty

Worse Cov

Aggre Unc

Worse Unc

(c) CIFAR10N

between 2 and 4 for CIFAR10N. For CIFAR100's IDN benchmarks, uncertainty is between 20 and 30. These results suggest that our prior clean label distribution is effective at selecting the correct clean label while reducing the number of label candidates during training.

Training time comparison One of the advantages of our approach is its efficient training algorithm, particularly when compared with other generative and discriminative methods. Tab. 7 shows the training time for competing approaches on CIFAR100 with 50% IDN and Clothing1M using the hardware specified in Sec. 4.2 . In general, our method has a smaller training time than competing approaches, being 1 . 4 × faster than CausalNL [50], 3 × faster than DivideMix [22], and and 13 × faster than InstanceGM [11].

5 Conclusion

In this paper, we presented a new learning algorithm to optimise a generative model represented by p ( X | Y ) that directly associates data and clean labels instead of maximising the joint data likelihood, denoted by p ( X, ˜ Y ) . Our optimisation implicitly estimates p ( X | Y ) with the discriminative model q ( Y | X ) eliminating the inefficient generative model training. Furthermore, we introduce an informative label prior for maintaining high coverage of latent clean label and regularise noisy label training. Results on synthetic and real-world noisy-label benchmarks show that our generative method has SOTA results, but with complexity comparable to discriminative models.

A limitation of the proposed method that needs further exploration is a comprehensive study of the model for q ( Y | X ) . In fact, the competitive results shown in this paper are obtained from fairly standard models for q ( Y | X ) without exploring sophisticated noisy-label learning techniques. In the future, we will use more powerful models for q ( Y | X ) . Another issue of our model is the difficulty to estimate p ( X | Y ) in real-world datasets containing images of high resolution. We will study more adequate ways to approximate p ( X | Y ) in such scenario using data augmentation strategies to increase the scale of the dataset.

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

329

330

331

332

333

334

335

336

337

338

339

340

341

342

343

344

345

346

347

348

349

350

351

References

E. Arazo, D. Ortego, P. Albert, N. O'Connor, and K. McGuinness. Unsupervised label noise modeling and loss correction. In International conference on machine learning , pages 312-321. PMLR, 2019.

D. Arpit, S. Jastrz˛ ebski, N. Ballas, D. Krueger, E. Bengio, M. S. Kanwal, T. Maharaj, A. Fischer, A. Courville, Y. Bengio, et al. A closer look at memorization in deep networks. In Proceedings of the 34th International Conference on Machine Learning-Volume 70 , pages 233-242. JMLR. org, 2017.

H. Bae, S. Shin, B. Na, J. Jang, K. Song, and I.-C. Moon. From noisy prediction to true label: Noisy prediction calibration via generative model, 2022.

P. Chen, B. B. Liao, G. Chen, and S. Zhang. Understanding and utilizing deep neural networks trained with noisy labels. In International Conference on Machine Learning , pages 1062-1070. PMLR, 2019.

Y. Chen and et al. Boosting co-teaching with compression regularization for label noise. In CVPR , pages 2688-2692, 2021.

D. Cheng and et al. Instance-dependent label-noise learning with manifold-regularized transition matrix estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 16630-16639, 2022.

H. Cheng, Z. Zhu, X. Li, Y. Gong, X. Sun, and Y. Liu. Learning with instance-dependent label noise: A sample sieve approach. In International Conference on Learning Representations , 2021.

A. P. Dempster, N. M. Laird, and D. B. Rubin. Maximum likelihood from incomplete data via the em algorithm. Journal of the Royal Statistical Society: Series B (Methodological) , 39(1):1-22, 1977.

J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition , pages 248-255. Ieee, 2009.

J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 , 2018.

A. Garg, C. Nguyen, R. Felix, T.-T. Do, and G. Carneiro. Instance-dependent noisy label learning via graphical modelling. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages 2288-2298, 2023.

B. Han, Q. Yao, X. Yu, G. Niu, M. Xu, W. Hu, I. Tsang, and M. Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. In Advances in neural information processing systems , pages 8527-8537, 2018.

K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learningfor image recognition. ComputerScience , 2015.

L. Jiang, D. Huang, M. Liu, and W. Yang. Beyond synthetic noise: Deep learning on controlled noisy labels. In International Conference on Machine Learning , pages 4804-4815. PMLR, 2020.

L. Jiang, Z. Zhou, T. Leung, L.-J. Li, and L. Fei-Fei. Mentornet: Learning data-driven curriculum for very deep neural networks on corrupted labels. In International Conference on Machine Learning , pages 2304-2313. PMLR, 2018.

T. Kaiser, L. Ehmann, C. Reinders, and B. Rosenhahn. Blind knowledge distillation for robust image classification. arXiv preprint arXiv:2211.11355 , 2022.

T. Kim, J. Ko, J. Choi, S.-Y. Yun, et al. Fine samples for learning with noisy labels. Advances in Neural Information Processing Systems , 34:24137-24149, 2021.

D. P. Kingma. Variational inference & deep learning: A new synthesis. 2017. 352

J. M. Köhler, M. Autenrieth, and W. H. Beluch. Uncertainty based detection and relabeling of noisy image 353 labels. In CVPR Workshops , pages 33-37, 2019. 354

A. Krizhevsky, G. Hinton, et al. Learning multiple layers of features from tiny images. 2009. 355

A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural 356 networks. Communications of the ACM , 60(6):84-90, 2017. 357

J. Li and et al. Dividemix: Learning with noisy labels as semi-supervised learning. ICLR , 2020. 358

X. Li, T. Liu, B. Han, G. Niu, and M. Sugiyama. Provably end-to-end label-noise learning without anchor 359 points. arXiv preprint arXiv:2102.02400 , 2021. 360

361

362

363

G. Litjens, T. Kooi, B. E. Bejnordi, A. A. A. Setio, F. Ciompi, M. Ghafoorian, J. A. Van Der Laak, B. Van Ginneken, and C. I. Sánchez. A survey on deep learning in medical image analysis. Medical image analysis , 42:60-88, 2017.

364

365

366

367

368

369

370

371

372

373

374

375

376

377

378

379

380

381

382

383

384

385

S. Liu, J. Niles-Weed, N. Razavian, and C. Fernandez-Granda. Early-learning regularization prevents memorization of noisy labels, 2020.

S. Liu, J. Niles-Weed, N. Razavian, and C. Fernandez-Granda. Early-learning regularization prevents memorization of noisy labels. Advances in neural information processing systems , 33:20331-20342, 2020.

Y. Liu, H. Cheng, and K. Zhang. Identifiability of label noise transition matrix. arXiv preprint arXiv:2202.02016 , 2022.

M. Lukasik, S. Bhojanapalli, A. K. Menon, and S. Kumar. Does label smoothing mitigate label noise? In International Conference on Machine Learning , 2020.

J. Lv, M. Xu, L. Feng, G. Niu, X. Geng, and M. Sugiyama. Progressive identification of true labels for partial-label learning. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event , volume 119 of Proceedings of Machine Learning Research , pages 6500-6510. PMLR, 2020.

E. Malach and S. Shalev-Shwartz. Decoupling" when to update" from" how to update". Advances in neural information processing systems , 30, 2017.

D. Ortego, E. Arazo, P. Albert, N. E. O'Connor, and K. McGuinness. Multi-objective interpolation training for robustness to label noise. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 6606-6615, 2021.

G. Patrini, A. Rozza, A. Krishna Menon, R. Nock, and L. Qu. Making deep neural networks robust to label noise: A loss correction approach. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 1944-1952, 2017.

E. Rolf, N. Malkin, A. Graikos, A. Jojic, C. Robinson, and N. Jojic. Resolving label uncertainty with implicit posterior models. arXiv preprint arXiv:2202.14000 , 2022.

Y. Shen and S. Sanghavi. Learning with bad training data via iterative trimmed loss minimization. In 386 International Conference on Machine Learning , pages 5739-5748. PMLR, 2019. 387

388

389

H. Song, M. Kim, and J.-G. Lee. Selfie: Refurbishing unclean samples for robust deep learning. In International Conference on Machine Learning , pages 5907-5915. PMLR, 2019.

Y. Tian, X. Yu, and S. Fu. Partial label learning: Taxonomy, analysis and outlook. Neural Networks , 2023. 390

391

392

H. Wang, R. Xiao, Y. Li, L. Feng, G. Niu, G. Chen, and J. Zhao. Pico+: Contrastive label disambiguation for robust partial label learning, 2022.

393

394

395

396

397

398

399

400

401

402

403

404

X. Wang, Y. Peng, L. Lu, Z. Lu, M. Bagheri, and R. M. Summers. Chestx-ray8: Hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 2097-2106, 2017.

Y. Wang, W. Liu, X. Ma, J. Bailey, H. Zha, L. Song, and S.-T. Xia. Iterative learning with open-set noisy labels. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 8688-8696, 2018.

H. Wei, L. Feng, X. Chen, and B. An. Combating noisy labels by agreement: A joint training method with co-regularization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 13726-13735, 2020.

J. Wei, H. Liu, T. Liu, G. Niu, and Y . Liu. Understanding (generalized) label smoothing whenlearning with noisy labels. arXiv preprint arXiv:2106.04149 , 2021.

J. Wei and Y. Liu. When optimizing f-divergence is robust with label noise. In International Conference 405 on Learning Representation , 2021. 406

407

408

409

410

411

412

413

414

415

416

417

418

419

420

421

422

423

424

425

426

427

428

429

430

431

432

433

434

435

436

437

438

439

440

J. Wei, Z. Zhu, H. Cheng, T. Liu, G. Niu, and Y. Liu. Learning with noisy labels revisited: A study using real-world human annotations. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022 . OpenReview.net, 2022.

X. Xia, T. Liu, B. Han, N. Wang, M. Gong, H. Liu, G. Niu, D. Tao, and M. Sugiyama. Part-dependent label noise: Towards instance-dependent label noise. Advances in Neural Information Processing Systems , 33:7597-7610, 2020.

X. Xia, T. Liu, N. Wang, B. Han, C. Gong, G. Niu, and M. Sugiyama. Are anchor points really indispensable in label-noise learning? Advances in Neural Information Processing Systems , 32, 2019.

T. Xiao, T. Xia, Y. Yang, C. Huang, and X. Wang. Learning from massive noisy labeled data for image classification. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 2691-2699, 2015.

Y. Xu, P. Cao, Y. Kong, and Y. Wang. L_dmi: A novel information-theoretic loss function for training deep nets robust to label noise. In Advances in Neural Information Processing Systems , pages 6225-6236, 2019.

Y. Xu, L. Zhu, L. Jiang, and Y. Yang. Faster meta update strategy for noise-robust deep learning. In CVPR , pages 144-153, 2021.

S. Yang, E. Yang, B. Han, Y. Liu, M. Xu, G. Niu, and T. Liu. Estimating instance-dependent bayes-label transition matrix using a deep neural network. In K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvári, G. Niu, and S. Sabato, editors, International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA , volume 162 of Proceedings of Machine Learning Research , pages 25302-25312. PMLR, 2022.

Y. Yao, T. Liu, M. Gong, B. Han, G. Niu, and K. Zhang. Instance-dependent label-noise learning under a structural causal model. Advances in Neural Information Processing Systems , 34:4409-4420, 2021.

T. Young, D. Hazarika, S. Poria, and E. Cambria. Recent trends in deep learning based natural language processing. ieee Computational intelligenCe magazine , 13(3):55-75, 2018.

F. Zhang, L. Feng, B. Han, T. Liu, G. Niu, T. Qin, and M. Sugiyama. Exploiting class activation value for partial-label learning. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022 . OpenReview.net, 2022.

H. Zhang, M. Cisse, Y. N. Dauphin, and D. Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint arXiv:1710.09412 , 2017.

Y. Zhang, S. Zheng, P. Wu, M. Goswami, and C. Chen. Learning with feature-dependent label noise: A progressive approach. arXiv preprint arXiv:2103.07756 , 2021.

Z. Zhu, T. Liu, and Y. Liu. A second-order approach to learning with instance-dependent label noise. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1011310123, 2021.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel approach to tackle the challenge of noisy label learning through a generative framework. Firstly, it presents a new model optimization technique that establishes a direct association between the data and clean labels. Secondly, the generative model is implicitly estimated by leveraging a discriminative model, thereby eliminating the need for training a separate generative model and enhancing efficiency. Thirdly, the paper proposes an informative label prior inspired by partial label learning, which serves as a supervision signal for noisy label learning. Extensive experiments conducted on various noisy-label benchmarks demonstrate that the proposed generative model achieves state-of-the-art results. Remarkably, it achieves these results while maintaining a comparable computational complexity to discriminative models.\n\nSTRENGTHS:\n- This paper introduces an informative prior for the latent clean label in noisy label learning, which is interesting.\n- Experimental results show the effectiveness of the proposed method.\n\nWEAKNESSES:\n1. My concern is whether it is reasonable to build a generative noisy-label learning model, which only assumes that Y causes X. The previous work, where the latent feature Z and Y cause X, seems more reasonable.\n2. From the perspective of the algorithm, it seems unnecessary to limit the algorithm to image data, while the notation at Line 138, Page 4 and the experiments focus on the image data, which is required to be explained or conduct more experiments on other kinds of datasets.\n3. Some symbols need to be explained. For example, at Eq.(9), Page 5, $y_i(j)$ and $p_i(j)$ is very confusing. So is $|\\mathcal{Y}|$ at Eq.(12), Page 5.\n4. Some paragraphs, especially for the approach, need to be polished up to better explain how to address the proposed three issues in the abstract.\n\nQUESTIONS:\nWhat is the practical scene of such a generative noisy-label learning model?\n\nLIMITATIONS:\nThe authors analyze the limitations in the conclusion of the paper. It is suggested that more explanations should be made on the reasonability of their generative model. Besides, some paragraphs need to be polished up.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper discusses the solution of learning with noisy labels by directly optimizing P(X|Y) relying on associating the data with clean labels directly. A informative label prior is derived with experimental results on several benchmark datasets.\n\nSTRENGTHS:\nThe author derive a solution for generative noisy label learning under some very strong and unrealistic assumptions.\n\nWEAKNESSES:\n(1)\tThe paper is very difficult to understand partially because the method is not well motivated. For example, there are many generative model based noisy label learning methods such as\n(i)\tLabel-Noise Robust Generative Adversarial Networks, CVPR 2019\n(ii)\tFrom Noisy Prediction to True Label: Noisy Prediction Calibration via Generative Model, ICML 2022.\nMoreover, the approaches that donot directly optimize the P(X|Y) usually because it is not tractable under general conditions. This paper is adding many strong but unrealistic assumptions to directly optimize P(X|Y) may not sound like a reasonable approach. Namely the contribution point 1 is not clear.\n(2)\tI have strong doubt about the theoretical soundness of the paper. For instance, in the equation (10), why the clean label prior can be defined as the linear combination of the three terms \\tilde{y}_i, c(i) and u_{i}(j), as these three terms may not be mutually exclusive and they may overlap with each other. \n\n(3)\tAnother issue is in the equation (12), the authors said “ the label ui is obtained by sampling from a uniform distribution of all possible labels proportionally to its probability of representing a noisy-label sample”, this assumption is way too strong and cannot be true. As a matter of fact, there are very few cases that the label distributions are uniform. Frequently, the noisy label distributions are highly imbalanced. Please take a look at the reference [ii] in ICML 2022, the usual way to assume noisy label distribution is multinomial instead of uniform distribution.\n\n(4)\tThe contribution is not clear. Besides the contribution 1 is not really a contribution, in the second point of their claimed contribution, they said “Our generative model is implicitly estimated with a discriminative model, making it computationally more efficient than previous generative approaches.”\nThis is also not valid as there are many generative adversarial network based noisy labeling which applied both generative model and discriminative model and let them collaborate with each other (for instance, the reference [i]) Thus, the second point is also very ordinary and not novel as well.\n\nQUESTIONS:\nThere are many issues in this paper including poor theoretical ground, unrealistic and wrong assumption as well as very limited novelty.\n\nLIMITATIONS:\nAs mentioned before, as the contribution of this paper is not clear. Both of the points (1) and (2) are actually done by previous work with a wider scope and less strict assumption.\n\nIn addition to that, assuming the noisy label distribution to be uniform is too restrictive, making the solution has little usage.\n\nLast but not least, the solution is derived based on very strong and unrealistic assumption such as equations (10) and (12), making the experimental results not convincing.\n\nIn summary, the paper is too far away from the level of a Neurips paper.", "TITLE": "Official Review", "RECOMMENDATION": 1, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 1, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on improving the efficiency of the generative model in the context of learning noisy labels. To achieve this, the authors first introduce a generative framework whose loglikelihood given a variational posterior can be extended into a label transition term and two KL-divergence terms. Then, the authors demonstrate that the optima of one of the KL-divergence terms could be used to transform a discriminative model into an implicit generative model without extra computation cost. \n\nTo further improve the performance of the proposed framework, the authors also present an informative label prior which combines benefits from both the high coverage (sample from Categorical distribution) and the low uncertainty (sample from Uniform distribution) as well as the information contains in the noisy labels.\n\nThe proposed work has two main contributions: 1) derive a KL-divergence term from the generative model that allows a discriminative model to be transformed into an implicit generative model, which guarantees the performance of a generative model and the efficiency of a discriminative model at the same time for learning noisy labels; 2) propose a novel clean label prior which allows a tradeoff between the label coverage and uncertainty. Although the second contribution is not as novel in terms of its simplicity, the visualization of its coverage and uncertainty enhances its impact.\n\nThe results of the wide range of experiments on benchmark datasets for noisy labels demonstrate the effectiveness of the proposed framework. The ablation studies are also carefully designed and conducted to validate the contribution of each component of the framework. \n\n###########################################################################\n\n######################### Post Rebuttal ######################################\n\n###########################################################################\n\nThe authors have addressed all of my concerns. I am happy to raise my score from 6 to 7.\n\nSTRENGTHS:\nThe paper is well-presented and easy to follow, with detailed descriptions of each component. The ablation of the framework and the analysis for the proposed clean prior are well performed. The performance gains on CIFAR benchmarks are significant, especially when the instance-dependent noise ratio is high. This indicates that the proposed model could indeed improve the model's robustness against noisy labels.\n\nMinimizing the KL-divergence term KLD( q(y|x) || p(x|y)p(y) ) to estimate the generative model parameters using discriminative model parameters is novel. In addition, the contribution regarding the clean label prior makes this paper a valuable addition to the community.\n\nWEAKNESSES:\nMinor:\n\n1. The authors should elaborate more on how the modeling P(X|Y) contributes to the informativeness of noisy labels in the introduction part.\n\n2. Please check the references. Some of them are incomplete (e.g., no journal or conference name for [37]).\n\nQUESTIONS:\nIn the related work, the authors mention that the estimation of the label transition matrix is troublesome. Based on equation (7), the estimation of the label transition matrix is still required for the proposed framework. I think the authors did not mention how their optimization of the label transition matrix (the cross-entropy term) differed from the existing work. So, how does the proposed framework alleviate the problems faced by label transition estimation?\n\nLIMITATIONS:\nNone", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

comments: SUMMARY:
Most previous works address learning with noisy labels with discriminative models while this paper takes the generative approach which maximizes directly on associated data and clean labels. This generative model is implicitly estimated with a discriminative model, making it computationally more efficient.

STRENGTHS:
- This paper successfully addresses several issues that might exist for approaches rely on generative models e.g. challenging to train and tend to use uninformative clean label priors.

- The experiments are extensive with results on many datasets with both realistic and synthetic label noise. All results show the effectiveness of the proposed method.

WEAKNESSES:
- The method is not well motivated. Even though most previous methods often adopt discriminative models and generative models are less discussed, it is still very hard for the reviewer to understand why generative is better than discriminative models for noisy label problems. 

- The author argues that the small loss hypothesis offers little guarantee of successfully selecting clean-label samples, however, this hypothesis is very related to the early-learning phenomenon in which the clean labels tend to fit earlier in the training than the noisy labels. This paper still uses this early learning to estimate the clean label prior. 

- Figure 2 is difficult to read and it does not make understanding the method any easier, the presentation of the paper should be significantly improved.

QUESTIONS:
- How does this generative method deal with the case when input $X$ is very large?

- Is there any intuition why the reversed KL loss eventually produces better results?

LIMITATIONS:
The authors adequately addressed the limitations.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07