Title: Rethinking Open-set Noise in Learning with Noisy Labels
Venue: NeurIPS 2024
Accepted: no
Keywords: Open-set noise

Noisy labels
Abstract: To reduce reliance on labeled data, learning with noisy labels (LNL) has gained increasing attention. However, prevailing works typically assume that such datasets are primarily affected by closed-set noise (where the true/clean labels of noisy samples come from another known category), and ignore therefore the ubiquitous presence of open-set noise (where the true/clean labels of noisy samples may not belong to any known category).
In this paper, we formally refine the LNL problem setting considering the presence of open-set noise. We theoretically analyze and compare the effects of open-set noise and closed-set noise, as well as the effects between different open-set noise modes. We also analyze common open-set noise detection mechanisms based on prediction entropy values. To empirically validate the theoretical results, we construct two open-set noisy datasets - CIFAR100-O/ImageNet-O and introduce a novel open-set test set for the widely used WebVision benchmark. Our work suggests that open-set noise exhibits qualitatively and quantitatively distinct characteristics, and how to fairly and comprehensively evaluate models in this condition requires more exploration.

Abstract

To reduce reliance on labelled data, learning with noisy labels (LNL) has gained increasing attention. However, prevailing works typically assume that such datasets are primarily affected by closed-set noise (where the true/clean labels of noisy samples come from another known category), and ignore therefore the ubiquitous presence of open-set noise (where the true/clean labels of noisy samples may not belong to any known category). In this paper, we formally refine the LNL problem setting considering the presence of open-set noise. We theoretically analyze and compare the effects of open-set noise and closed-set noise, as well as the effects between different open-set noise modes. We also analyze common open-set noise detection mechanisms based on prediction entropy values. To empirically validate the theoretical results, we construct two open-set noisy datasets - CIFAR100-O/ImageNet-O and introduce a novel open-set test set for the widely used WebVision benchmark. Our work suggests that open-set noise exhibits qualitatively and quantitatively distinct characteristics, and how to fairly and comprehensively evaluate models in this condition requires more exploration.

1 Introduction

In recent years, the tremendous success of machine learning often relies on the assumption that data 17 labels are accurate and free from noise. However, in real-world scenarios, label noise caused by 18 factors such as annotation errors and label ambiguity is ubiquitous, posing a pervasive challenge to 19 the performance and generalization of models. To address this challenge, various methods have been 20 proposed to learn with noisy labels, including noise transition matrix [7, 23], label correction [17, 3], 21 robust loss functions [6, 29, 19], and recently dominant sample selection-based approaches [11, 2]. 22

23

24

25

26

27

28

29

Most current efforts, however, primarily focus on closed-set noise, where the true labels of noisy samples belong to another known class. This includes common noise models like symmetric noise (assuming that the labels of samples are randomly flipped with a certain probability to any other known classes) or asymmetric noise model (assuming that the probability of label confusion is influenced by the classes, such as 'cat' being more likely to be confused with 'dog' than with 'airplane'). Recent advancements have also explored instance-dependent noise models [4, 26], where label confusion depends directly on individual instances.

Unfortunately, unlike the in-depth exploration of closed-set noise, there is noticeably limited research 30 on open-set noise, where the true labels of noisy samples may not belong to any known category. 31 This gap becomes particularly crucial when considering one of the primary motivations for learning 32 with noisy labels: learning with datasets obtained through web crawling. Examining one of the most 33 commonly used benchmarks - the WebVision dataset [12], we validate the prevalence of open-set 34 noise (fig. 1). In fact, the 'open-world' assumption involving open-set samples has received more 35 attention in other weakly supervised learning problems, such as open-set recognition and outlier 36

37

38

39

40

41

42

43

44

45

46

47

48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

Tench?

detection, but lacks enough exploration in the context of LNL. To this end, we focus on a thorough theoretical analysis of open-set noise in this paper. Specifically:

Considering the presence of open-set noise, we introduce the concept of a complete noise transition matrix and reformulate the LNL problem and label noise definition in this context.

To enable offline analysis, we consider two pragmatic cases: fitted case , that the model perfectly fits the noisy distribution, and memorized case , that the model completely memorises the noisy labels.

We analyze and compare the open-set noise vs. closed-set noise on closed-set classification accuracy and suggest that open-set noise has a less negative impact in both cases. We also analyze and compare the 'hard' open-set noise vs. 'easy' open-set noise, but find that these two different noise modes show opposite trends in two different cases.

Since closed-set classification evaluation may be insufficient to fully reflect model performance, we consider introducing an additional open-set detection task and conduct preliminary experiments.

We derive and analyze the open-set noise detection mechanism based on the entropy values of model predictions and suggest that it may be effective only for 'easy' open-set noise. We also consider two representative LNL methods and combine them with such open-set noise detection mechanism for further experiments.

For controlled experiments, we construct two novel synthetic open-set noise datasets: CIFAR100-O and ImageNet-O. Additionally, we introduce a new open-set test set to the WebVision dataset for the open-set detection task.

2 Related works

Methods for learning with noisy labels can be roughly categorized into two main directions. The first direction typically focuses on estimating noise transition matrix [4, 26, 23, 7] or designing robust loss functions [29, 19, 6], aiming to achieve theoretically risk-consistent or probabilistic-consistent models. However, most of these works often assume an ideal scenario where the model can learn to fit the sampled distribution well, overlooking the over-fitting issues arising from excessive model capacity and insufficient data in practical situations. In this paper, we introduce the concept of complete noise transition matrix considering the presence of open-set noise and conduct theoretical analyses and experimental validations for both ideal case and over-fitting case, namely fitted case and memorized case . The second type is often based on sample selection strategies, involving also different regularization terms and off-the-shelf techniques such as semi-supervised learning and model co-training, to achieve the state-of-the-art performance. Most sample selection methods are based on the model's current predictions, such as the popular 'small loss' mechanism [2, 11, 8, 28, 10, 17, 13, 27, 24, 30], or model's feature space [21, 22, 15, 5].

Especially, the investigation on open-set noise is relatively scarce. Wang et al. [18] utilize Local 72 Outlier Factor algorithm to identify open-set noise in feature space, Wu et al. [22] propose to identify 73 open-set noise with subgraph connectivity, while both Sachdeva et al. [16] and Albert et al. [1] try to 74 identify open-set noise based on entropy-related dynamics. Instead, Feng et al. [5] do not identify 75 open-set noise explicitly while avoid relabelling and including open-set noise in the training. More 76 closely related to our work, Xia et al. [25] also investigates noise transition matrices involving open77 set noise but considering all open-set noise belonging to a single meta-class. In this paper, we consider 78

that open-set noise may originate from different classes, and based on this premise, we analyze two 79 distinct open-set noise modes. Wei et al. [20] propose leveraging open-set noise to mitigate the 80 impact of closed-set noise, as it helps alleviating the model's over-fitting tendency. Instead, we focus 81 on a thorough theoretical analysis of the effects with different noise modes, including open-set noise 82 versus closed-set noise, and different open-set noise versus each other. 83

84

85

86

87

88

89

90

91

92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

3 Methodology

In section 3.1, we briefly introduce the traditional problem formulation of LNL. In section 3.2, we reformulate the LNL problem considering open-set noise. In section 3.3, we formalize how label noise influences model generalization, particularly, on the proposed error rate inflation metric. In section 3.4, we analyze and compare the impact of open-set vs. closed-set noise, as well as 'easy' open-set noise vs. 'hard' open-set noise. In section 3.5, we scrutinize the open-set noise detection mechanism based on model prediction entropy values.

3.1 Traditional formulation of LNL

Supervised classification learning typically assumes that we sample a certain number of independently and identically distributed training samples { x k , y k } K k =1 from a joint distribution P ( x , y ; y ∈ Y in ) , i.e., the so-called train set. By default, here all the possible values for y k in the discrete label space Y in : { 1 , 2 , ..., A } (referred here as inlier classes ), are known in advance. With a certain loss function, given the train set { x k , y k } K k =1 we aim to train a model f : x → y whose predictions can achieve the minimum error rate under the whole clean distribution P ( x , y ; y ∈ Y in ) .

Under LNL problem setting, we believe that the joint distribution P ( x , y ; y ∈ Y in ) has been perturbed to P n ( x , y ; y ∈ Y in ) ; especially, the conditional distribution P n ( y | x ; y ∈ Y in ) changes - normally we assume the sampling prior is free of the label noise ( P ( x ; y ∈ Y in ) = P n ( x ; y ∈ Y in )), leading to the presence of noisy labels y n k in the noisy train set { x k , y n k } K k =1 that do not conform to the clean conditional distribution P ( y | x ; y ∈ Y in ) .

3.2 Revisiting LNL considering open-set noise

We here formally revisit the problem formulation of learning with noisy labels considering the existence of open-set noise. Instead of assuming all the possible classes are known (y ∈ Y in ), we consider samples from some unknown outlier classes may also exist in the train set. Let us denote these classes as outlier classes Y out : { A +1 , A +2 , ..., A + B } with B as the number of possible outlier classes. Then, we expand the support of joint distribution to contain both inlier and outlier classes, denoted as P ( x , y ; y ∈ Y in ∪ Y out ) and P n ( x , y ; y ∈ Y in ∪ Y out ) for the clean and noisy ones, respectively. For brevity, we denote as Y all ≜ Y in ∪Y out . Similarly as above, we still assume the noisy labelling will not affect the sampling prior ( P ( x ; y ∈ Y all ) = P n ( x ; y ∈ Y all ) ). For subsequent analysis, we first define below complete noise transition matrix:

Definition 3.1 (Complete noise transition matrix) . For a specific sample x , we define as T (sample index omitted here for simplicity) the complete noise transition matrix 1 :

T in corresponds to the confusion process between inlier classes Y in : { 1 , 2 , ..., A } , and T out 115 corresponds to the confusion process from outlier classes Y out : { A +1 , A +2 , ..., A + B } to inlier 116 classes Y in : { 1 , 2 , ..., A } . 117

For brevity, we denote as T ij ≜ P ( y n = j | y = i, x = x ; y n , y ∈ Y all ) . We have further 118 ∑ A + B j =1 T ij = 1 for i ∈ { 1 , ..., A + B } - noise transition from each clean class sums to 1 over all 119 possible noisy classes. With such a complete noise transition matrix T , we can connect the clean 120

conditional distribution P ( y | x = x ; y ∈ Y all ) with the noisy conditional distribution P n ( y | x = 121 x ; y ∈ Y all ) as below: 122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

Label noise Recent works usually discriminate label noise into closed-set noise and open-set noise. Before continuing with the further discussion, we feel it is necessary to elucidate these two concepts here clearly to avoid any ambiguities, as we will try to comparably discriminate and analyze them later. Specifically, most of recent works define open-set noise as 'a sample with its true label from unknown classes but mislabelled with a known label'. Formally, we have:

Definition 3.2 (Label noise) . For sample x with clean label y and noisy label y n :

When y = y n , ( x , y, y n ) is a clean sample;

̸

When y = y n and y ∈ Y in , ( x , y, y n ) is a closed-set noise;

̸

When y = y n and y ∈ Y out , ( x , y, y n ) is an open-set noise.

Specifically, we have y ∼ P ( y = y | x = x ; y ∈ Y all ) while y n ∼ P n ( y = y n | x = x ; y ∈ Y all ) .

However, we can only identify label noise type with ( x , y, y n ) -y, y n yet to be sampled even with known conditional probability. To enable sample-wise analysis on the impact of different label noise, we further introduce below ( O x , C x ) label noise :

Definition 3.3 (( O x , C x ) label noise) . For sample x with clean conditional probability P ( y | x = x ; y ∈ Y all ) and complete noise transition matrix T :

̸

Here, O x is the expected open-set noise ratio, C x is the expected closed-set noise ratio. We then define sample x as an ( O x , C x ) label noise. Intuitively speaking, sample x is expected to be an open-set noise with probability as O x and to be a closed-set noise with probability C x .

With Definition 3.3, we formalize the concept of noise ratio for the whole distribution, as the accumulated ( O x , C x ) label noise at all sample points x ∈ X :

3.3 Analyzing classification error rate inflation in LNL

In this section, we try to analyze the impact of different label noise. Please note, while the reformulated LNL setting encompasses outlier classes Y out , in both the training and evaluation stage, they are unknown (agnostic); the learned model f is still tailored for the classification of inlier classes Y in . That is to say, the default classification evaluation protocol is still concerned with the classification error rate over the inlier conditional probability, denoted as P f ( y | x = x ; y ∈ Y in ) .

Error rate inflation With P f ( y | x = x ; y ∈ Y in ) , in the evaluation phase, for specific sample x we have its prediction as: y f = arg max k P f ( y = k | x = x ; y ∈ Y in ) ∈ Y in , and the corresponding expected classification error rate as:

̸

Specifically, we have the Bayes error rate corresponds to the Bayes optimal model f ∗ : 152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

To measure the negative impacts of noisy labels, we care about how much extra errors have been introduced, measured by the error rate inflation of learned model f compared to the Bayes optimal model f ∗ :

Definition 3.4 (Error rate inflation) . With E ∗ x as the Bayes error rate, we define the error rate inflation for sample x as: ∆ E x = E x -E ∗ x .

Two pragmatic cases However, P f ( y | x = x ; y ∈ Y in ) , as the prediction of the final learned model f , is affected by many factors (model capacity/dataset size/training hyperparameters such as training epochs, etc.), which is non-trivial to determine its specific value for an offline analysis 2 . Thus, we consider two specific pragmatic cases:

Fitted case : the model perfectly fits the noisy distribution: P f ( y | x = x ; y ∈ Y in ) = P n ( y | x = x ; y ∈ Y in ) ;

Memorized case : the model completely memorises the noisy labels: P f ( y | x = x ; y ∈ Y in ) = P y n ( y | x = x ; y ∈ Y in ) ; Here P y n denotes the one-hot encoding of the noisy label y n .

Nonetheless, these two cases are very realistic and important; Empirically, it is highly possible that the memorized case can correspond to scenarios such as scratch training based on a single-label dataset with a normal deep neural network - as normally such model has enough capacity to memorize all the labels, while the fitted case can correspond to scenarios such as fine-tuning a linear classifier with a pre-trained model - as the pre-trained model already captures good sample representations and the capacity of a linear classifier is limited.

3.4 Error rate inflation analysis w.r.t different label noise

In this section, we focus on analyzing the error rate inflation of different label noise. Let us recall the clean conditional distribution as P ( y | x ; y ∈ Y all ) . For ease of analysis, we contemplate a simple scenario, wherein the entire clean conditional distribution remains unchanged, except only one of the sample points, say x , is afflicted by label noise:

̸

̸

̸

In this condition, we can simplify analyzing the impact of label noise on the whole distribution to analyzing the error rate inflation of a single sample x . Specifically, we consider two specific sample points x 1 and x 2 , corresponding to two in our later comparative analysis. Let us denote its clean conditional probability as P ( y | x = x 1 ; y ∈ Y all ) = [ p 1 1 , ..., p 1 A , ..., p 1 A + B ] and P ( y | x = x 2 ; y ∈ Y all ) = [ p 2 1 , ..., p 2 A , ..., p 2 A + B ] , and noise transition matrix as T 1 and T 2 , respectively. We further assume:

We compare the error rate inflation ( ∆ E x 1 vs ∆ E x 2 ) with different label noise given same/fixed noise ratio for a strictly fair comparison. Note we assume that x 1 and x 2 hold the same sampling prior probability: P ( x = x 1 ; y ∈ Y all ) = P ( x = x 2 ; y ∈ Y all ) ); so that, we assure that the whole noise ratio N is fixed, and more importantly, sample x 1 and x 2 can be considered as probabilistic exchangeable in the dataset collection process.

For better clarity, we depict the derivation relations for ∆ x in fig. 2. Specifically, for our two interested cases above, we have corresponding error rate inflation for sample x (sample subscript omitted for simplicity) as:

Fitted case :

Memorized case :

We notice that ∆ x in both cases are only affected by clean conditional probability P ( y | x = x 1 ; y ∈ 193 Y all ) and complete noise transition matrix T . 194

195

196

197

198

199

200

201

202

203 204

205

206

207

208

209

210

211

212

213

214

215

Clean conditional probability:

Noisy conditional probability:

Complete Noise Transition Matrix:

Noisy conditional probability over

inlier classes:

Clean conditional probability over

inlier classes:

Error Rate Inflation:

C

eq.(2)

T

eq. (1)

P(y｜x=c;y ∈yall)

eq.(4)

Pn(y丨x =c;y ∈ in)

O

eq. (2)

P(y丨x=a;y ∈yall)

eq.(3)

P(y丨x=;y ∈yin)

eq. (7)

3.4.1 How does open-set noise compare to closed-set noise?

We first try to elucidate the difference between open-set noise and closed-set noise. Without loss of generality, we consider:

Intuitively speaking, we consider sample x 1 to be more prone to open-set noise compared to sample x 2 , thus corresponding to the 'more open-set noise' scenario. However, without extra regularizations, there exist infinite T 1 and T 2 fulfilling eq. (7) and eq. (10) given specific P ( y | x = x 1 ; y ∈ Y all ) and P ( y | x = x 2 ; y ∈ Y all ) (see toy example below), the analysis on ∆ E x 1 vs ∆ E x 2 is thus infeasible.

Toy example about agnostic T Assuming a ternary classification, with two known inlier classes ('0" and '1") and one unknown outlier class '2". Say, we have sample x 1 with clean conditional probability as [0 . 1 , 0 . 2 , 0 . 7] . Assuming two different noise transition matrices for T 1 below:

We have O x 1 = 0 . 7 , C x 1 = 0 . 2 in both conditions but we arrive at different noisy conditional probability, similarly for sample x 2 .

̸

We thus consider a class concentration assumption - in most classification datasets, the majority of samples belong to specific class exclusively with high probability. In this condition, we have proved: Theorem 3.5 (Open-set noise vs closed-set noise) . Let us consider sample x 1 , x 2 fulfilling eq. (7) and eq. (10) - compared to x 2 , x 1 is considered as more prone to open-set noise. Let us denote a = arg max i P ( y = i | x = x 1 ; y ∈ Y all ) and b = arg max i P ( y = i | x = x 2 ; y ∈ Y all ) , we assume (with a high probability): p 1 a → 1 , { p 1 i → 0 } i = a and p 2 b → 1 , { p 2 b → 0 } i = b . Then, we have:

̸

in both Fitted case and Memorized case .

Please refer to appendix D.1 for detailed proof. To summarize, we validate that in most conditions, open-set noise is less harmful than closed-set noise in both fitted case and memorized case .

3.4.2 How does different open-set noise compare to each other?

We further study how different open-set noise affect the model. Specifically, we consider:

Intuitively speaking, we focus on the impacts of different open-set noise modes given the same/fixed open-set noise ratio, while excluding the effect of closed-set noise. In this section, we assume sample x 1 and sample x 2 holds the same clean conditional probability: [ p 1 1 , ..., p 1 A , ..., p 1 A + B ] = [ p 2 1 , ..., p 2 A , ..., p 2 A + B ] , to only focus on the impact of different open-set noise modes with the same original sample. It is straightforward that O x 1 = O x 2 always holds since ∑ A + B i = A +1 p 1 i = ∑ A + B i = A +1 p 2 i . To ensure C x 1 = C x 2 = 0 , we simply set T 1 in = T 2 in = I .

216

217

218

219

220

221

and 222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

Thus, we have the flexibility to explore various forms of T out -corresponding to different open-set noise modes. Specifically, we consider two distinct open-set noise modes: 'easy' open-set noise when the transition from outlier classes to inlier classes involves completely random flipping, and 'hard' open-set noise when there exists an exclusive transition between the outlier class and specific inlier class. We denote as T easy for 'easy' open-set noise and T hard for 'hard' open-set noise, with intuitive explanations below:

Especially, for T easy , we have T ij = 1 A everywhere; for T hard , we denote as H i : { arg j ( T hard ji = 1) } A i =1 the set of corresponding outlier classes j ∈ Y out confused to inlier class i ∈ Y in . Without loss of generality, we consider x 1 with 'easy' open-set noise T easy and x 2 with 'hard' open-set noise T hard . Please note, that we no longer require class concentration assumption here as the noise transition matrix is already known. In this condition, we have proved:

Theorem 3.6 ('Hard' open-set noise vs 'easy' open-set noise) . Let us consider sample x 1 , x 2 fulfilling eq. (7) and eq. (11) . We set the corresponding noise transition matrix as T 1 out = T easy , T 2 out = T hard , T 1 in = T 2 in = I and denote [ p 1 1 , ..., p 1 A , ..., p 1 A + B ] = [ p 2 1 , ..., p 2 A , ..., p 2 A + B ] = [ p 1 , ..., p A , ..., p A + B ] . Then, we have:

· Fitted case :

· Memorized case :

Please refer to appendix D.2 for detailed proof. Specifically, we further discuss about memorized case here. Since ∑ A i =1 b i = 0 , ∑ A i =1 a i = 1 , we can easily infer max(∆ E x 1 -∆ E x 2 ) ≥ 0 , min(∆ E x 1 -∆ E x 2 ) ≤ 0 . With theorem D.3, we know when the ranking of { p 1 i } A i =1 is completely in agreement with the ranking { ∑ j ∈ H i p 1 j } A i =1 (constant term -1 A ∑ A + B i = A +1 p 1 i omitted here), we reach its maximum value with ∆ E x 1 -∆ E x 2 ≥ 0 . Intuitively speaking, this implies a scenario that the 'hard' open-set noise tends to confuse a sample into the inlier class it primarily belongs to (with higher semantic similarity), as indicated by its higher probability (the higher the p 1 i the higher the ∑ j ∈ H i p 1 j ). For example, an outlier 'tiger' image is wrongly included as a 'cat' rather than a 'dog' in a 'cat vs dog' binary classification dataset. As this is more consistent with the common intuition, we default to such noise mode for 'hard' open-set noise - assuming the ranking of { p 1 i } A i =1 is of high agreement with the ranking of { ∑ j ∈ H i p 1 j } A i =1 .

To summarize, unlike the general comparison between open-set noise and closed-set noise, the 'hard' open-set noise and the 'easy' open-set noise exhibit an opposite trend in two different cases. In the fitted case , 'easy' open-set noise appears to be less harmful, while in the memorized case , the impact of 'hard' open-set noise is comparatively smaller.

3.5 Rethinking open-set noise detection

In this section, we try to investigate a commonly used open-set noise identification mechanism based on entropy dynamics. Within the sample selection paradigm, several methods [1, 16] have proposed to further identify open-set noise, based on the empirical phenomenon that samples with relatively in-confident predictions are usually open-set samples, characterized by its high prediction entropy. Specifically, we consider original sample x without noise transition, x with T hard and x with T easy

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

270

271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

as a clean sample, a 'hard' open-set noise and an 'easy' open-set noise, respectively. For simplicity, we omit the subscript.

Empirically, most sample selection method starts from the early training stages after certain epochs of warm-up training, expecting the model to learn meaningful information before over-fitting. To analyze the entropy dynamics, we thus consider the model predictions in the fitted case as a pragmatic proxy. Let us denote as H easy , H hard and H clean the prediction entropy corresponds to these three conditions, we have 3 :

We note H easy ≥ H clean 4 . However, comparing H hard and H clean is non-trivial without specific values for each entry. Thus, we suggest open-set noise detection based on the prediction entropy may only be effective for 'easy' open-set noise .

4 Experiments

In this section, we try to validate our theoretical findings. In section 4.1, we validate the theoretical comparisons of different label noise. In section 4.2, we validate the entropy dynamics with different label noise. Moreover, in appendix E.1, we revisit the performance of two existing LNL methods involving open-set noise. To conduct more controllable, fair and accurate experiments, we propose two synthetic open-set noisy datasets - CIFAR100-O and ImageNet-O, respectively based on the CIFAR100 and ImageNet datasets. We also consider closed-set noise in some experiments, particularly, the symmetric closed-set noise. Please refer to appendix A for more dataset and implementation details and also details about open-set detection protocol.

4.1 Empirical validation on previous probabilistic findings

In this section, we conduct experiments to validate the theorem 3.5 and theorem 3.6. Since most deep models have sufficient capacity, we consider direct supervised learning from scratch on the noisy dataset and consider the final model as the memorized case - as evidenced by nearly 100% train set accuracy. Conversely, obtaining a model that perfectly fits the data distribution is often challenging; here, we consider training a single-layer linear classifier upon a frozen pretrained encoder. Due to the limited capacity of the linear layer, we expect to roughly approach the fitted case .

We show classification accuracy on CIFAR100-O and ImageNet-O datasets under different noise ratios, as shown in fig. 3(a/b). We find that: 1) in both cases, the presence of open-set noise has a significantly smaller impact on classification accuracy compared to closed-set noise. 2) 'hard' open-set noise and 'easy' open-set noise show opposite trends in the two different scenarios. These results align perfectly with our theoretical analysis.

In addition to closed-set classification accuracy, we also report the model's open-set detection performance using the maximum prediction value as the indicator [9]) in fig. 3(c/d). We find that, in both cases, the presence of open-set noise leads to a degraded open-set detection performance, while conversely, the presence of closed-set noise can often even enhance open-set detection performance. In light of this contrasting trend, we propose that the open-set detection task, in addition to the default closed-set classification, may help to offer a more comprehensive evaluation of LNL methods.

291

292

293

294

295

296

297

(a) Closed-set classification in

fitted case

(b) Closed-set classification in

memorized case

(c) Open-set detection in

fitted case

(d) Open-set detection in

memorized case

4.2 Inspecting entropy-based open-set noise detection mechanism

In section 3.5, we briefly analyze the open-set detection mechanism based on the entropy values of model predictions and find that it may be effective only for 'easy' open-set noise. Here, we again utilize the CIFAR100-O and ImageNet-O datasets for validation experiments with different open-set noise ratios and modes. Specifically, we adopt the common warm-up idea used in existing LNL methods - training with the entire dataset for a certain number of epochs. We report the model's predicted entropy values for each sample at the { 5 th, 10 th, 20 th, 30 th } epoch in fig. 4.

Epoch 10

Epoch 20

Epoch 30

Epoch 10

Epoch 20

Epoch 30

Epoch 10

Epoch 20

Epoch 30

Epoch 10

Epoch 20

Epoch 30

(a) CIFAR100-O with 20% 'easy' open-set noise

(c) ImageNet-O with 40% 'easy' open-set noise

(b) CIFAR100-O with 20% 'hard' open-set noise

(d) ImageNet-O with 40% 'hard' open-set noise

We validate that the entropy dynamics is a more effective indicator for 'easy' open-set noise compared 298 to 'hard' open-set noise ((a) vs (b), (c) vs (d) in fig. 4). However, even for 'easy' open-set noise, we 299 also notice that the warm-up epoch matters a lot - too early ( 5 th epoch in fig. 4(c)) or too late ( 30 th 300 epoch in fig. 4(c)) also make open-set noise difficult to distinguish. We also test with mixed noise 301 including both open-set noise and closed-set noise, please refer to appendix B for more discussions. 302

303

5 Conclusions

This paper focuses on exploring how open-set label noise affects the performance of models. While 304 the 'open world' setting involving open-set samples has been widely discussed in several other weakly 305 supervised learning settings, its application in the context of learning with noisy labels has been 306 understudied. In light of this, we reconsider the LNL problem, specifically focusing on the impact of 307 open-set noise compared to closed-set noise, and different types of open-set noise compared to each 308 other, on the evaluation performance. In light of the challenges existing testing frameworks face in 309 handling open-set noise, we explore the open-set detection task to address the deficiencies in model 310 evaluation for open-set noise and conducted preliminary experiments. Additionally, we look into 311 the common mechanism for detecting open-set noise based on the model's prediction entropy. Both 312 theoretical and empirical results highlight the urgent need for a deeper exploration of open-set noise 313 and its complex impact on model performance. 314

References 315

Paul Albert, Diego Ortego, Eric Arazo, Noel E O'Connor, and Kevin McGuinness. Addressing 316 out-of-distribution label noise in webly-labelled data. In Proceedings of the IEEE/CVF Winter 317 Conference on Applications of Computer Vision , pages 392-401, 2022. 2, 7 318

Eric Arazo, Diego Ortego, Paul Albert, Noel O'Connor, and Kevin McGuinness. Unsupervised 319 label noise modeling and loss correction. In International Conference on Machine Learning , 320 pages 312-321. PMLR, 2019. 1, 2 321

Paola Cascante-Bonilla, Fuwen Tan, Yanjun Qi, and Vicente Ordonez. Curriculum labeling: 322 Revisiting pseudo-labeling for semi-supervised learning. In Proceedings of the AAAI Conference 323 on Artificial Intelligence , volume 35, pages 6912-6920, 2021. 1 324

Pengfei Chen, Junjie Ye, Guangyong Chen, Jingwei Zhao, and Pheng-Ann Heng. Beyond 325 class-conditional assumption: A primary attempt to combat instance-dependent label noise. In 326 Proceedings of the AAAI Conference on Artificial Intelligence , volume 35, pages 11442-11450, 327 2021. 1, 2 328

329

330

331

Chen Feng, Georgios Tzimiropoulos, and Ioannis Patras. Ssr: An efficient and robust framework for learning with unknown label noise. In 33rd British Machine Vision Conference 2022, BMVC 2022, London, UK, November 21-24, 2022 . BMVA Press, 2022. 2, 19, 20

Aritra Ghosh, Himanshu Kumar, and PS Sastry. Robust loss functions under label noise for deep 332 neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 31, 333 2017. 1, 2 334

Jacob Goldberger and Ehud Ben-Reuven. Training deep neural-networks using a noise adapta335 tion layer. 2016. 1, 2 336

337

338

339

Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. arXiv preprint arXiv:1804.06872 , 2018. 2

Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of-distribution 340 examples in neural networks. In International Conference on Learning Representations , 2016. 341 8, 13 342

Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, and Li Fei-Fei. Mentornet: Learning 343 data-driven curriculum for very deep neural networks on corrupted labels. In International 344 Conference on Machine Learning , pages 2304-2313. PMLR, 2018. 2, 12 345

Junnan Li, Richard Socher, and Steven CH Hoi. Dividemix: Learning with noisy labels as 346 semi-supervised learning. arXiv preprint arXiv:2002.07394 , 2020. 1, 2, 12, 19, 20 347

Wen Li, Limin Wang, Wei Li, Eirikur Agustsson, and Luc Van Gool. Webvision database: 348 Visual learning and understanding from web data. arXiv preprint arXiv:1708.02862 , 2017. 1, 349 12 350

Eran Malach and Shai Shalev-Shwartz. Decoupling" when to update" from" how to update". 351 arXiv preprint arXiv:1706.02613 , 2017. 2 352

Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar. Foundations of machine learning . 353 MIT press, 2018. 5 354

Diego Ortego, Eric Arazo, Paul Albert, Noel E O'Connor, and Kevin McGuinness. Multi355 objective interpolation training for robustness to label noise. In Proceedings of the IEEE/CVF 356 Conference on Computer Vision and Pattern Recognition , pages 6606-6615, 2021. 2, 12 357

Ragav Sachdeva, Filipe R Cordeiro, Vasileios Belagiannis, Ian Reid, and Gustavo Carneiro. 358 Evidentialmix: Learning with combined open-set and closed-set noisy labels. In Proceedings of 359 the IEEE/CVF Winter Conference on Applications of Computer Vision , pages 3607-3615, 2021. 360 2, 7, 12 361

Hwanjun Song, Minseok Kim, and Jae-Gil Lee. Selfie: Refurbishing unclean samples for robust 362 deep learning. In International Conference on Machine Learning , pages 5907-5915. PMLR, 363 2019. 1, 2 364

Yisen Wang, Weiyang Liu, Xingjun Ma, James Bailey, Hongyuan Zha, Le Song, and Shu-Tao 365 Xia. Iterative learning with open-set noisy labels. In Proceedings of the IEEE conference on 366 computer vision and pattern recognition , pages 8688-8696, 2018. 2 367

368

369

370

Yisen Wang, Xingjun Ma, Zaiyi Chen, Yuan Luo, Jinfeng Yi, and James Bailey. Symmetric cross entropy for robust learning with noisy labels. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 322-330, 2019. 1, 2

371

372

373

374

375

376

377

378

379

380

381

382

383

384

385

386

387

388

389

390

391

392

393

394

Hongxin Wei, Lue Tao, Renchunzi Xie, and Bo An. Open-set label noise can improve robustness against inherent label noise. Advances in Neural Information Processing Systems , 34:7978-7992, 2021. 3

Pengxiang Wu, Songzhu Zheng, Mayank Goswami, Dimitris Metaxas, and Chao Chen. A topological filter for learning with label noise. Advances in neural information processing systems , 33:21382-21393, 2020. 2

Zhi-Fan Wu, Tong Wei, Jianwen Jiang, Chaojie Mao, Mingqian Tang, and Yu-Feng Li. Ngc: A unified framework for learning with open-world noisy data. arXiv preprint arXiv:2108.11035 , 2021. 2, 12

Xiaobo Xia, Tongliang Liu, Nannan Wang, Bo Han, Chen Gong, Gang Niu, and Masashi Sugiyama. Are anchor points really indispensable in label-noise learning? Advances in neural information processing systems , 32, 2019. 1, 2

Xiaobo Xia, Tongliang Liu, Bo Han, Mingming Gong, Jun Yu, Gang Niu, and Masashi Sugiyama. Sample selection with uncertainty of losses for learning with noisy labels. arXiv preprint arXiv:2106.00445 , 2021. 2

Xiaobo Xia, Bo Han, Nannan Wang, Jiankang Deng, Jiatong Li, Yinian Mao, and Tongliang Liu. Extended T: Learning with mixed closed-set and open-set noisy labels. IEEE Transactions on Pattern Analysis and Machine Intelligence , 45(3):3047-3058, 2022. 2

Shuo Yang, Erkun Yang, Bo Han, Yang Liu, Min Xu, Gang Niu, and Tongliang Liu. Estimating instance-dependent bayes-label transition matrix using a deep neural network. In International Conference on Machine Learning , pages 25302-25312. PMLR, 2022. 1, 2

Kun Yi and Jianxin Wu. Probabilistic end-to-end noise correction for learning with noisy labels. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 7017-7025, 2019. 2

Xingrui Yu, Bo Han, Jiangchao Yao, Gang Niu, Ivor Tsang, and Masashi Sugiyama. How does 395 disagreement help generalization against label corruption? In International Conference on 396 Machine Learning , pages 7164-7173. PMLR, 2019. 2 397

Zhilu Zhang and Mert R Sabuncu. Generalized cross entropy loss for training deep neural 398 networks with noisy labels. arXiv preprint arXiv:1805.07836 , 2018. 1, 2 399

Tianyi Zhou, Shengjie Wang, and Jeff Bilmes. Robust curriculum learning: from clean label de400 tection to noisy label self-correction. In International Conference on Learning Representations , 401 2020. 2 402

403

404

405

406

407

408

409

410

411

412

413

414

415

416

417

418

419

420

421

422

423

424

425

426

427

428

429

430

431

432

433

434

435

436

437

438

439

440

441

442

443

444

445

446

447

448

A Experiment details

A.1 Dataset details

Previous works involving open-set noise also try to build synthetic noisy datasets, typically treating different datasets as open-set noise for each other to construct synthetic noisy dataset [16, 22]. In this scenario, potential domain gaps could impact a focused analysis of open-set noise. In this work, we propose selecting inlier/outlier classes from the same dataset to avoid this issue. Besides, in previous works, the consideration of open-set noise patterns often focused on random flipping from outlier classes to all possible inlier classes, which is indeed the 'easy' open-set noise adopted here. However, both our theoretical analysis and experimental findings demonstrate that 'easy' open-set noise and 'hard' open-set noise exhibit distinct characteristics. Therefore, relying solely on experiments with 'Easy' open-set noise is insufficient, emphasizing the necessity to explore and understand the complexities associated with different types of open-set noise. We also evaluate with closed-set noise in some experiments, by default, we consider the common symmetric closed-set noise in this work.

CIFAR100-O For the original CIFAR100 dataset, in addition to the commonly-used 100 fine classes, there exist 20 coarse classes each consisting of 5 fine classes. To build CIFAR100-O, we select one fine class from each coarse class as an inlier class (20 classes in total) while considering the remaining classes as outlier classes (80 classes in total). Then, we consider 'Hard' and 'Easy' open-set noise as below:

'Hard': Randomly selected samples from the same coarse category as the target category were introduced as open-set noise.

'Easy': Regardless of the target category, samples from the remaining categories were randomly introduced as open-set noise.

ImageNet-O For a more challenging benchmark, we consider ImageNet-1K datasets - consisting of 1,000 classes. Specifically, we randomly select 20 classes and artificially identify another 20 classes similar to each of them:

inliers= ['tench', 'great white shark', 'cock', 'indigo bunting', 'European fire salamander', 'African

outliers= ['goldfish', 'tiger shark', 'hen', 'robin', 'common newt', 'American alligator', 'garden 'academic gown', 'beer glass', 'bookshop', 'CD player', 'coffeepot', 'shopping cart', 'ice cream', 'pizza']

Then, we consider 'Hard' and 'Easy' open-set noise as below:

'Hard': Randomly select samples from the corresponding similar outlier class as the target category were introduced as open-set noise.

'Easy': Samples from the remaining categories were randomly introduced as open-set noise.

For open-set detection, we directly use the corresponding test sets of these classes from the original datasets.

WebVision WebVision [12] is an extensive dataset comprising 1,000 classes of images obtained through web crawling, which thus contains a large amount of open-set noise. In line with previous studies [10, 11, 15], we evaluate our methods using the first 50 classes from the Google Subset of WebVision. To test the performance of open-set detection on the WebVision dataset, we collect a separate test set consisting of open-set images, following the same collection process as the WebVision dataset. Specifically, we utilize the Google search engine with the class names as keywords and identify those open-set samples that haven't been included in the train set for this test set.

A.2 Implementation details 449

450

451

Here, we provide detailed implementation specifications for the fitted case and memorized case in section 4.1. We also briefly the applied open-set detection protocol.

452

453

454

455

456

457

458

459

460

461

462

463

464

465

466

467

468

469

470

471

472

473

474

475

476

477

478

479

480

481

Fitted case For the fitted case , we train a randomly initialized classifier - a single linear layer based on the encoder of the ResNet18 model with pretrained weights. In the case of the CIFAR100-O dataset, a weak augmentation strategy involving image padding and random cropping is applied during training, with a batch size of 512. The weight decay (wd) is set to 0.0005, and the model undergoes training for 100 epochs, utilizing a learning rate (lr) of 0.02. The learning rate schedule follows a cosine annealing strategy.

For the ImageNet-O dataset, no augmentation is applied during training. The batch size is maintained at 512, with a weight decay (wd) of 0.01. The model is trained for 100 epochs, employing a learning rate (lr) of 0.02. The learning rate schedule for this case also adheres to a cosine annealing strategy.

Memorized case In this case, we train a PreResNet18 model from scratch. For both datasets, a weak augmentation strategy involving image padding and random cropping is applied during training, with a batch size of 128. The weight decay (wd) is set to 0.0005, and the model undergoes training for 200 epochs, utilizing a learning rate (lr) of 0.02. The learning rate schedule also follows a cosine annealing strategy.

Open-set detection protocol We use the maximum softmax probability in [9] for the open-set detection task. Specifically, assume the trained model f outputs a softmax vector p i for each sample x i . We then choose a threshold value t between 0 and 1. For evaluation, we consider binary labels indicating whether a sample belongs to a known class (closed-set) or the open-set and convert the open-set detection task into a binary classification problem. Samples with a maximum softmax value p max i below the threshold are considered potential open-set samples. This is because a low maximum value indicates the model is less confident in any specific class for that sample.

B Entropy dynamics for mixed label noise

In addition to the open-set noise only scenario, we also inspect the entropy dynamics with mixed label noise in fig. 5. Here, we use the notation '0.2all_0.5easy' to represent a scenario where the total noise ratio is 0.2, and within this, half of them are 'easy' open-set noise. In the presence of mixed label noise, the existence of closed-set noise severely interferes with identifying open-set noise. For example, in fig. 5(d), the entropy values of open-set noise even exceed those of clean samples. Though not theoretically analyzed, this further suggests that entropy dynamics based on model predictions, may be fragile, and we need to handle open-set noise more cautiously.

Epoch 10

Epoch 20

Epoch 30

(a) CIFAR100-O 0.2all_0.5easy

Epoch 10

Epoch 20

Epoch 30

(b) CIFAR100-O 0.2all_0.5hard

Epoch 10

Epoch 20

Epoch 30

Epoch 10

Epoch 20

Epoch 30

(c) ImageNet-O 0.4all_0.5easy

(d) ImageNet-O 0.4all_0.5hard

C Error rate inflation in two different cases

In this section, we present the computation details of error rate inflation in two interested cases fitted 482 case and memorized case . Specifically, we have: 483

· Fitted case : 484

While E ∗ x denotes the Bayes optimal error rate: 486

We thus have ∆ E x in both cases as: 487

· Fitted case : 488

Memorized case : 489

Details on the derivation of error rate inflation (fig. 2) Then, we describe the essential concepts 490 depicted in fig. 2 in detail. For better clarity, we here restate the notations in section 3.4. We explicitly 491 consider two specific sample points x 1 and x 2 being perturbed independently, corresponding to two 492 different label noise modes. Let us assume its clean conditional probability as: 493

and denote its noise transition matrix as T 1 = { T 1 ij } A + B i,j =1 and T 2 = { T 2 ij } A + B i,j =1 , respectively. Here, 494 { T 1 ij = 0 } , { T 2 ij = 0 } for all j > A . 495

With eq. (1), we compute the corresponding noisy conditional probability for both samples as: 496

Note that the error rate inflation is dependent on the clean conditional probability over inlier classes , 497 noisy conditional probability over inlier classes and sampling prior over inlier classes as shown in 498 eq. (18) and eq. (19). 499

Specifically, for sample x 1 , we have: 500

Simply changing the subscript leads us to the formulations for sample x 2 . To summarize, wrapping 501 the above together, we have: 502

We here omit the sample subscript and abbreviate the proportional symbol for simplicity. With 503 eq. (18), eq. (19) and eq. (23), we can then compute and compare ∆ E x in both fitted case and 504 memorized case : 505

506

507

508

509

510

511

which leads us to:

̸

D Full proof of theorem 3.5 and theorem 3.6

Error rate inflation comparison s.t. same noise ratio To ensure a fair comparison, in this work, we focus on the impact of different label noise given the same noise ratio - modifying O x and C x while analyzing the trend of ∆ E x . Specifically, for above mentioned x 1 and x 2 , we further assume:

̸

Please note, here the clean conditional probability is considered as known and fixed, while eq. (27) 512 restricts the values of the noise transition matrix T 1 and T 2 , given specific clean conditional 513 probability. We then analyze and compare the error rate inflation in both conditions. 514

515

516

517

518

519

520

521

522

523

D.1 Proof of theorem 3.5 - Open-set noise vs Closed-set noise

In this section, we try to compare open-set noise and closed-set noise. Without loss of generality, we consider:

Intuitively speaking, sample x 1 is more affected by open-set noise compared to sample x 2 , thus corresponding to the interested 'open-set noise'.

As clarified by the toy example in section 3.4.1, without extra regularizations, the noise transition matrix is not identifiable. We thus consider a simple compromise situation - in most classification problems, the majority of samples (with a high probability) belong to a specific class exclusively with high probability.

Let us denote:

and

We assume :

and we have:

̸

With eq. (28), we easily infer that: a ∈ Y out while b ∈ Y in . Intuitively speaking, x 1 is an open-set 524 noise, with its clean conditional probability concentrated on one of the outlier classes, and vice versa 525 for x 2 . 526

With eq. (27), we further have:

̸

Thus we have: T 2 bb ≈ 0 , which enables us to analyze and compare ∆ E x 1 and ∆ E x 2 : 527

Fitted case In this case, according to eq. (24), we have: 528

̸

529

̸

̸

̸

Memorized case In this case, according to eq. (25), we similarly have: 530

We wrap up above for theorem D.2: 531

Theorem D.1 (Open-set noise vs Closed-set noise) . Let us consider sample x 1 , x 2 fulfilling eq. (26) and eq. (28) - compared to x 2 , x 1 is considered as more prone to open-set noise. Let us denote a = arg max i P ( y = i | x = x 1 ; y ∈ Y all ) and b = arg max i P ( y = i | x = x 2 ; y ∈ Y all ) , we assume (with a high probability): p 1 a → 1 , { p 1 i → 0 } i = a and p 2 b → 1 , { p 2 b → 0 } i = b . Then, we have:

̸

in both fitted case and memorized case . 532

533

D.2 Derivation of theorem 3.5 - 'hard' open-set noise vs 'easy' open-set noise

In this part, we try to analyze and compare 'hard' open-set noise with 'easy' open-set noise. For 534 better clarification, we repeat here the essential statements: 535

Especially, for T easy , we have T ij = 1 A everywhere; for T hard , we denote as H i : { arg j ( T hard ji = 1) } A i =1 the set of corresponding outlier classes j ∈ Y out confused to inlier class i ∈ Y in . We also have:

and 536

and 537

. 538

Fitted case In this case, according to eq. (24), for sample x 1 with 'easy' open-set noise, we have: 539

and, for sample x 2 with 'hard' open-set noise, we have: 540

̸

Memorized case In this case, according to eq. (25), for sample x 1 with 'easy' open-set noise, we 541 have: 542

and, for sample x 2 with 'hard' open-set noise, we have: 543

Let a i = p 1 i , b i = ∑ j ∈ H i p 1 j -1 A ∑ A + B i = A +1 p 1 i , we have:

To summarize, we wrap up the above together:

544

Theorem D.2 ('Hard' open-set noise vs 'easy' open-set noise) . Let us consider sample x 1 , x 2 545 fulfilling eq. (26) and eq. (11) . We set the corresponding noise transition matrix as in eq. (33) , eq. (34) 546 and eq. (35) . We further assume [ p 1 1 , ..., p 1 A , ..., p 1 A + B ] = [ p 2 1 , ..., p 2 A , ..., p 2 A + B ] . Then, we have: 547

in memorized case . Here, a i = p 1 i , b i = ∑ j ∈ H i p 1 j -1 A ∑ A + B i = A +1 p 1 i . 548

Theorem D.3 (Rearrangement Inequality) . For the sequences a 1 , a 2 , . . . , a n and b 1 , b 2 , . . . , b n , where a 1 ≤ a 2 ≤ . . . ≤ a n and b 1 ≤ b 2 ≤ . . . ≤ b n , the rearrangement inequality is given by:

Here, σ denotes a permutation of the indices 1 , 2 , . . . , n . The leftmost expression corresponds to the case where σ ( i ) = i (identity permutation), and the rightmost expression corresponds to the case where σ ( i ) = n +1 -i (reverse permutation).

E Revisiting LNL methods

E.1 Revisiting existing LNL methods with open-set noise

In this section, we further investigate the learning effectiveness of existing LNL methods on previously discussed open-set label noise, especially the dominant ones based on sample selection - these methods often integrate different regularization terms and off-the-shelf techniques, resulting in state-of-the-art performance. In essence, such methods typically include a sample selection module along with a

549

550

551

552

553

554

555

556

557

We further have:

in fitted case ,

robust training module. Here, we briefly denote the clean subset selected by the original method 558 as X clean and denote the entire dataset as X all . Moreover, we consider integrating the previously 559 mentioned open-set detection mechanism into current LNL methods - we denote as X in an inlier 560 subset based on entropy dynamics. Then, maintaining the robust training module unchanged, we 561 consider below three different variants (the involved LNL method abbreviated as X , the inlier subset 562 detection method abbreviated as EntSel ): 563

564

565

566

X: Robust training using X clean , i.e., the original method;

EntSel: Robust training using X in ;

X + EntSel: Robust training using X in ∩ X clean .

Specifically, we test with two representative LNL methods with well-maintained open-source imple567 mentations: SSR [5] and DivideMix [11]. Please refer to appendix E.2 for more details. In fig. 6, we 568 show results on CIFAR100-O and ImageNet-O. 569

0.2

0.2

0.4

0.4

0.2

0.2

0.4

0.4

Easy

Hard

r

a

r

o

0.2

0.2

0.4

0.4

0.2

0.2

0.4

0.4

Easy

Hard

r

a

r

o

0.2

0.2

0.4

0.4

0.2

0.2

0.4

0.4

Easy

Hard

r

a

r

o

0.2

0.2

0.4

0.4

0.2

0.2

0.4

0.4

Easy

Hard

r

a

r

o

0.2

0.2

0.4

0.4

0.2

0.2

0.4

0.4

Easy

Hard

r

a

r

o

0.2

0.2

0.4

0.4

0.2

0.2

0.4

0.4

Easy

Hard

r

a

r

o

0.2

0.2

0.4

0.4

0.2

0.2

0.4

0.4

Easy

Hard

r

a

r

o

0.2

0.2

0.4

0.4

0.2

0.2

0.4

0.4

Easy

Hard

r

a

r

o

First, focusing on the classification accuracy of the model, we observe that 1) using EntSel instead of 570 the original method leads to a reduction in classification accuracy in the mixed noise scenario (SSR 571 vs EntSel and DivideMix vs EntSel); in pure open-set noise only scenarios, there are no obvious 572 trends showing differences in different variant models. 2) the classification accuracy for mixed noise 573 is significantly lower than that of only open-set noise at the same noise ratio, which further confirms 574 that closed-set noise is more harmful than open-set noise. 575

Furthermore, we demonstrate the performance of this model in detecting open-set samples - the 576 introduction of EntSel significantly enhances the effectiveness of open-set detection, especially 577 when the open-set noise is set to 'easy' mode. This also further confirms our theoretical analysis in 578 section 3.5 and experimental results in section 4.2.

580

581

582

583

584

585

586

587

588

589

590

591

592

593

594

595

596

597

598

599

600

601

602

603

604

605

606

607

608

609

610

611

612

613

614

615

616

617

618

619

620

621

622

623

624

625

626

627

628

We report results for the WebVision dataset in table 1, reaffirming that combining 'EntSel' with 'SSR' significantly enhances open-set detection performance. Notably, most open-set noise in WebVision seems to arise from factors like text co-occurrence rather than semantic similarity, categorizing it more as 'easy' open-set noise. This may explain why EntSel effectively improves open-set detection in this context. However, when combining EntSel with DivideMix, both classification accuracy and open-set detection decrease, indicating that the robustness of the EntSel method itself is questionable. Additionally, simply merging SSR/DivideMix with EntSel using subset intersection (X + EntSel) also leads to a decrease in both classification accuracy and open-set detection performance. Finally, it's worth mentioning that, despite having lower classification accuracy than SSR, DivideMix outperforms SSR in open-set detection ROC AUC scores. All above illustrates that simply evaluating the classification accuracy may be one-sided.

E.2 Details of involved methods

DivideMix [11] Denoting as L = { l i } N i =1 the losses of all samples, DivideMix proposes to model it (after min-max normalization) with a Gaussian Mixture Model. The probabilities { p i } N i =1 of each sample belonging to the component with a smaller mean value are then extracted. Samples with probability p i greater than the threshold θ are then identified as a 'clean" subset. Link to code: https://github.com/LiJunnan1992/DivideMix .

SSR [5] In contrast to DivideMix, SSR extracts features for each sample and constructs a neighbourhood graph. By computing the nearest neighbour labels for each sample, a pseudo-label distribution p is obtained through a KNN voting process. The consistency c = p y / p max between this voted distribution and the given noisy label y (logit label) is then calculated. Samples with consistency c greater than the threshold θ are identified as part of the 'clean" subset. Link to code: https://github.com/MrChenFeng/SSR_BMVC2022 .

EntSel Wealso provide a concise overview of the steps involved in EntSel, following a methodology similar to DivideMix. Denoting as E = { e i } N i =1 the entropy of all samples' predictions, we similarly model it (after min-max normalization) with a Gaussian Mixture Model. The probabilities { p i } N i =1 of each sample belonging to the component with a smaller mean value are then extracted. Samples with probability p i greater than the threshold θ ′ are then identified as 'inlier" subset.

Generally, we have a closed-set classifier g and an encoder f, and we use it for training based on the selected subset. Existing sample selection methods usually rely on an estimated prediction and a threshold to help filter clean samples. Our proposed OpenAdaptor focuses on the difference between open-set and closed-set samples. When integrating them, we propose two different strategies: absorption and exclusion.

E.3 Implementation details

Experiment details For both SSR and DivideMix, we employ model and optimization configurations on the same dataset. Specifically, for CIFAR100-O and ImageNet-O, we utilize the PresResNet18 model, trained for 300 epochs with a batch size of 128 and a learning rate of 0.02, and a cosine annealing schedule was implemented. For the WebVision dataset, we utilize the ResNet18 model, training for 120 epochs with a reduced batch size of 32. The learning rate is set to 0.01 and controlled by a cosine annealing scheduler too. Additionally, a warm-up training phase of 10 epochs is implemented in the CIFAR100-O and ImageNet-O experiments, while a 5-epoch warm-up training phase is utilized in the WebVision experiment.

Hyperparameters In all experiments, we set the sample selection threshold θ ′ = 0 . 5 for EntSel. For SSR, we employ a sample selection threshold θ = 1 . 0 in all experiments. For DivideMix, the sample selection threshold remains constant at θ = 0 . 5 across all experiments. Both SSR and DivideMix incorporate MixUp, and we adhere to the original paper's choices by setting the MixUp coefficient to 4 for experiments on CIFAR100-O and ImageNet-O and to 0.5 for experiments on WebVision. Please note, as exploring and comparing these methods are not our focus, we believe there exist better hyperparameter settings.

648

′

Robustness of EntSel Asmaller θ for EntSel leads to better performance on WebVision - especially 629 when EntSel is used with DivideMix. If we set θ ′ = 0 . 2 , our classification accuracy increases from 630 62.96% to 67.2%, ROC AUC increases from 0.8166 to 0.8599 (table 1). However, we are keen to use 631 fixed hyperparameters in all experiments as we emphasize that the hyperparameter robustness is also 632 critical for LNL methods. 633

634

F More examples of open-set noise in WebVision dataset

In this section, we present additional examples of open-set noise within the 'Tench' class of the 635 WebVision dataset. We trace the origin of web pages containing some open-set noise images. 636 Remarkably, we identify that the appearance of the term 'Tench' or related keywords is prevalent 637 on the web pages hosting these open-set noise images. We posit that this occurrence is attributed 638 to the data collection process on the web. Specifically, in the course of keyword searches and 639 crawling for images, instances were inadvertently included due to the presence of keywords in image 640 descriptions or accompanying text, such as people with 'tench' in the name, or related fishing tools. 641 As highlighted earlier, the prevalent belief in the current LNL community is that real-world noise 642 primarily arises from confusion induced by semantic similarity. Consequently, numerous recent 643 studies have concentrated on instance-dependent noise and related theoretical analysis. However, our 644 findings here indicate that in real-world scenarios, particularly in web-crawled datasets, noise may 645 be unrelated to semantics but instead caused by other latent high-dimensional information, such 646 as accompanying text here. Addressing such real-world noise requires increased attention and 647 further exploration.

0lv77S5PaW5vlM.jpg

http://ukscblog.com/ussc-v-uksc/

fIfzvcQnea4zsM.jpg

m59l4cCfDgox6M.jpg

X051MdLtrztWqM.jpg

usK0NyB0Q9VsVM.jpg

https://charlyanderic.travellerspoint.com/67/

https://www.drennantackle.com/drennan-distance-specialist-tench-bream-12ft-2lb-rod/

https://en.wikipedia.org/wiki/You_Should_Be_So_Lucky

http://pete777-pete777.blogspot.com/2012/04/tench-rigs-27th-april.html

Image

Source

USSCVUKSC

DAN TENCH NEWS ARTICLES

Yesterday's decision of the Us Supreme Court in NEIB

ef a/ y Sebelius regarding the Affordable Care Act

("ACA", aka Obamacare), which has convulsed the

American political scene, usefully lllustrates the

Important differences between the US Court and its UK

equivalent.

his

Reception

649

650

651

652

653

654

655

656

657

658

659

660

661

662

663

664

665

666

667

668

669

670

671

672

673

674

675

676

677

678

679

680

681

682

683

684

685

686

687

688

689

690

691

692

693

694

695

696

697

698

699

700

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We clearly and briefly describe our method and our contributions in the abstract and introduction sections.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]

Justification: In section 5 We specifically discuss the potential and limitations of current LNL method in learning with open-set noise.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We conduct theoretical analysis on the impact of open-set noise and provided a complete proof in appendix D.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: All the dataset and implementation details are included in appendix A.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: The complete codes will be released upon acceptance.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: All the dataset and implementation details are included in appendix A.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We conduct multiple runs and report averaged results in most experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: All experiments are conducted on a private server with 3 RX6000 GPUs.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We confirm that the conducted reserach conform with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: Not applicable.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Not applicable.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We include all essential information and references to the used datasets in this work.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

910

911

912

913

914

915

916

917

918

919

920

921

922

923

924

925

926

927

928

929

930

931

932

933

934

935

936

937

938

939

940

941

942

943

944

945

946

947

948

949

950

951

952

953

954

955

956

957

958

959

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: Not applicable.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: Not applicable.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: Not applicable.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper extends the problem setting of learning with noisy labels (LNL) to include open-set noise, where noisy labels may come from unknown categories, in contrast to the traditional focus on closed-set noise. The authors theoretically compare the impacts of open-set and closed-set noise and analyze detection mechanisms based on prediction entropy. They construct two open-set noisy datasets, CIFAR100-O and ImageNet-O, and introduce an open-set test set for the WebVision benchmark to validate their findings. Their results show that open-set noise exhibits distinct characteristics from closed-set noise. The paper emphasizes the need for comprehensive evaluation methods for models in the presence of open-set noise, calling for further research in this area.\n\nSTRENGTHS:\n- The research problem is interesting. Compared with learning with closed-set noise, learning with open-set noise is under-explored. \n- The theoretical analysis seems to be solid.\n\nWEAKNESSES:\n- Some technical details are hard to follow. Writing needs to be polished. \n- The contribution from the algorithm perspective is not enough.\n\nQUESTIONS:\n- I am a bit confused about the title of this work. This paper provides some insights into the research community. However, it does not provide some essentially different and surprising conclusions. Therefore, perhaps it is not suitable to use \"rethinking\". \n- The review part is not comprehensive. The methods combining unsupervised and semi-supervised learning are not reviewed and discussed. \n- The assumption that the noisy labeling will not affect the sampling prior is a bit strong. Could the paper supplement more intuitions about this assumption?\n- Will the size of label space for the open-set label, i.e., $B$, affect the theoretical analysis?\n- There is a gap between Line 165 and the following explanations, which makes it hard to understand. \n- Does Eq. (7) mean the bounded noise? \n- The method of predicting entropy values is similar to the methods in OOD detection. Could the method supplement more details? \n- The paper does not provide enough contribution from the algorithm perspective, such as how to estimate the transition matrix and how to better use/remove the open-set data for better model robustness. The harm of open-set label noise is less than closed-set one is known, which has been reflected in the previous works (results or related discussions).\n\nLIMITATIONS:\nN/A.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces an approach to address the challenge of open-set noise in the context of learning from noisy labels. The authors propose a method that differentiates between 'easy' and 'hard' types of open-set noise, which is critical for improving the robustness and performance of learning models faced with noisy data. By integrating existing Learning with Noisy Labels (LNL) techniques with novel entropy-based noise detection mechanisms, the paper presents both theoretical insights and empirical validations of the proposed methods. The contributions are significant as they offer a refined perspective on handling different noise complexities, which can enhance the utility of machine learning models in real-world applications dealing with noisy labels.\n\nSTRENGTHS:\nOriginality: The paper addresses the issue of open-set noise in learning from noisy labels with a novel approach, differentiating between 'easy' and 'hard' noise types. This nuanced consideration is original as it pushes the boundaries of how noise is typically treated in noisy label learning.\n\nQuality: The theoretical explanations are thorough and complemented by robust empirical evidence that strengthens the methodological claims.\n\nClarity: The paper is well-structured, offering clear explanations of complex concepts, which aids in understanding the proposed methods and their implications.\n\nSignificance: The significance of this work is evident as it tackles a critical issue that can potentially enhance model robustness and performance in real-world scenarios where label noise is common.\n\nWEAKNESSES:\nDependency on Specific Methods: The reliance on entropy-based techniques for noise distinction may not generalize across all scenarios or noise types.\n\nExperimental Scope: The experiments primarily utilize synthetic datasets, which might not fully capture the complexity of real-world data applications.\n\nQUESTIONS:\nRegarding Open-set Noise Types: How do 'easy' and 'hard' open-set noises specifically impact the robustness of different deep learning models across various architectures?\n\nLIMITATIONS:\nLimited Experimental Scope: The experimental validation focuses predominantly on synthetic datasets like CIFAR100-O and ImageNet-O. While these are commonly used in the research community for benchmarking, the real-world implications of the findings might be limited without additional testing on more varied and real-world datasets.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on open-set label noise problem. Authors first formally extending closed-set transition matrix to open-set transition matrix and define two noise ratios for open-set and closed-set separately. \nThen authors define error inflation rate as a measurement for noisy label impact and measure for two conditions, classifier fitted noisy distribution or memorized (overfit) noisy label. Later, authors propose a new type of open-set noise by exclusively transitioning outlier classes to a specific inlier class, and consider this as a \"hard\" open-set noise and traditional open-set noise as \"easy\" case. Authors further analysis two noise types on two classifier conditions and claim traditional entropy based open-set detection might only works on \"easy\" case. Experiments are performed on CIFAR-100, ImageNet and Webvision datasets.\n\nSTRENGTHS:\nAuthors formally define open-set noise with a similar symmetric/asymmetric setup as closed-set noise, and find out that it shows opposite trend with different classifier cases.\n\nWEAKNESSES:\n- The experiment parts lack of baselines. With a new type of noise proposed, previous baselines on easy open-set noise should be run to assess the performance gap and set up the benchmark.\n- Figure 4 (a) and (b) have similar distribution, it is hard to draw conclusions from entropy dynamics.\n- Supp E.1 results are confusing. \"X+EntSel\" should be a better strategy since it selects inlier clean samples. However, why the closed-set classification accuracy is always the worst? Table 1 Webvision result is similar as well. Why is the claim \"EntSel + SSR improves open-set detection performance\" valid? The Acc and AUC are both dropping after adding EntSel. Why SSR/DivideMix + EntSel is always the worst performance? Considering it is a combination of inlier and clean, shouldn't it be the best performing one? I assume this is still the normal accuracy and AUC, which is the higher the better.\n\nQUESTIONS:\n- what is the pre-trained weight fitted case used?\n\nLIMITATIONS:\nAuthors do not address any limitations in conclusion. A possible limitation might be related to approximation of fitted and memorized classifiers.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

comments: SUMMARY:
The paper refines the problem of learning with noisy labels (LNL) by addressing the often overlooked issue of open-set noise. It provides a comprehensive theoretical analysis comparing the impacts of open-set and closed-set noise, introduces novel datasets for empirical validation, and explores the effectiveness of entropy-based noise detection mechanisms.

STRENGTHS:
- The paper offers a thorough theoretical analysis of the differences between open-set and closed-set noise, extending the current understanding of LNL.
- The exploration of entropy-based mechanisms for detecting open-set noise adds a practical tool for improving LNL methods.
- This paper is well-written and easy to understand.

WEAKNESSES:
- The author summarizes two types of open-set noise, i.e., the easy and the hard noise, which is very similar to the symmetric and asymmetric label noise from the perspective of the transition matrix. So does there exist the instance-dependent open-set noise? What is its form if exists?

- In Section 3.5, the author conducts analyses regarding entropy dynamics-based open-set detection, which belongs to the **Fitted case**. If adopting the vision language model (such as CLIP) to fine-tune and detect the open-set noise, is it aligned with the **Memorized case**? It would be better for the author to provide a real-world application for the memorized case.

- The author should clearly illustrate the construct method of closed-set in the experiment (Figure 3) for reproducibility.

QUESTIONS:
See weaknesses

LIMITATIONS:
See weaknesses
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-15