Wasserstein Distributional Normalization : Nonparametric Stochastic Modeling for Handling Noisy Labels
Authors: Sung Woo Park
Junseok Kwon
Venue:  (2021)
Keywords: Wasserstein distributional normalization
Noisy labels
Classification

Abstract: We propose a novel Wasserstein distributional normalization (WDN) algorithm to handle noisy labels for accurate classification.  In this paper, we split our data into uncertain and certain samples based on small loss criteria. We investigate the geometric relationship between these two different types of samples and enhance this relation to exploit useful information, even from uncertain samples.  To this end, we impose geometric constraints on the uncertain samples by normalizing them into the Wasserstein ball centered on certain samples. Experimental results demonstrate that our WDN outperforms other state-of-the-art methods on the Clothing1M and CIFAR-10/100 datasets, which have diverse noisy labels. The proposed WDN is highly compatible with existing classification methods, meaning it can be easily plugged into various methods to improve their accuracy significantly. 

Under review as a conference paper at ICLR 2021
WASSERSTEIN DISTRIBUTIONAL NORMALIZATION NONPARAMETRIC STOCHASTIC MODELING FOR HAN- DLING NOISY LABELS
Anonymous authors Paper under double-blind review
ABSTRACT
We propose novel Wasserstein distributional normalization (WDN) algorithm to handle noisy labels for accurate classification. In this paper, we split our data into uncertain and certain samples based small loss criteria. We investigate the geometric relationship between these two different types of samples and enhance this relation to exploit useful information, even from uncertain samples. To this end, we impose geometric constraints on the uncertain samples by normalizing them into the Wasserstein ball centered on certain samples. Experimental results demonstrate that our WDN outperforms other state-of the-art methods on the Clothing M and CIFAR- datasets, which have diverse noisy labels. The proposed WDN is highly compatible with existing classification methods, meaning it can be easily plugged into various methods to improve their accuracy significantly.
INTRODUCTION
The successful results of deep neural networks (DNNs) on supervised classification tasks heavily rely on accurate and high-quality label information. However, annotating large-scale datasets is extremely expensive and a conssconn task. Because obtaining high-quality datasets very difficult, in most conventional works, training data have been obtained alternatively using crowd-sourcing platforms Yu et al. (2018) to obtain large-scaled datasets, which leads inevitable noisy labels in the annotated samples.
While there are numerous methods that can deal with noisy labeled data, recent methods actively adopt the small loss criterion, which enables to construct classification models that are not susceptible to noise corruption. In this learning scheme, a neural network trained using easy samples first the early stages of training. Harder samples then gradually selected to train mature models training proceeds. Jiang et al. (2018) suggested collaborative learning models, in which a mentor network delivers the data-driven curriculum loss student network. Haan et al. (2018); Yu al. (2019) proposed dual networks to generate gradient information jointly using easy samples and employed this information to allow the networks teach each other. Wei et al. (2020) adopted disagreement strategy, which determines the gradient information to update based on disagreement values between dual networks. Han et al. (2020) implemented accumulated gradients to escape optimization processes from over-paree parameterization and to obtain more generalized results. In this paper, we tackle to solve major issues raised from the aforementioned methods based on small-loss criterion, as follows.
In comprehensive experiments, the aforementioned methods gain empirical insight regarding network behavior under noisy labels. However, theoretical and quantitative explanation have not been closely investigated. In contrast, give strong thcoretical/empirical explanations to understand the network under noisy labels. In particular, we present an in-depth analysis of small loss criteria a probabilistic sense. We exploit the stochastic properties of noisy labeled data and develop probabilistic descriptions of data under the small loss criteria, as follows. Let IP be a probability measure for the pre-softmax logits of the training samples, be objective function for classification, and 1 (()) be indicator function. Then, our central object to deal with is a truncated measure defined as
X ~ ulsç = 1(xUX)x<) P
Y Y~ 1(xamss))
(1)
P((X) > (]
PI(Y) S

---

Under review as a conference paper at ICLR 2021
where X and Y, which are sampled from μl/ş and Els. denote uncertain and certain samples defined in the pre-softmax feature space (i.e., R²), respectively, resctivv In equation I and & denote the probability measures of uncertain and certain samples, respectively, and Ç is constant. Most previous works have focused on the usage of Y and the sampling strategy of ç. but poor generalization capabilities based the abundance of uncertain samples X has not been thoroughly investigated, even though these samples potentially contain important information. To understand the effect of noisy labels on the generalized bounds, we provide the concentration inequality of uncertain measure μ., which renders the probabilistic relation between μ and & and learnability of the network under noisy labels.
While most conventional methods Han et al. (2018); Wei et al. (2020): Li al. (2019a); Yu al.) (2019) require additional dual networks to guide misinformed noisy samples, the scalability is not guaranteed due to the existence of dual architectures, which have the same number of parameters as the base network. To alleviate this problem, we build statistical machinery, which should be fully non-parametric, simple implement, and computationally efficient reduce the computational complexity of conventional approaches, while maintaining the concept of small-loss criterion. Based on the empirical observation of ill-behaved certain/uncertain samples, we propose the gradient flow in the Wasserstein space, which can be induced by simulating aa-aaarmmetrcc stochastic differential equation (SDE) with respect to the -lenbbeek type to control the ill-behaved dynamics. The reason for selecting these dynamics will be thoroughly discussed in the following sections.
Thus, key contributions of our work are as follows.
We theoretically verified that there exists strong correlation between model confidence and statistical distance between X and Y. We empirically investigate that the classification accuracy worsens when the upper-bound of 2- 2-Wasserstein distance (2(,,)) {) S (i.e., distributional distance between certain and uncertain samples) drastically increase. Due to the empirical nature of upper-boume E, it can be used an estimator to determine if network suffers from over- wver-parameterization
Based on empirical observations, develop simple, non paraaaaaetri,c and computationally efficient stochastic model to control the observed ill-behaved sample dynamics. As a primal object, we propose the stochastic dynamics of gradient flow (i.e.,, Ornstein-Ulenbeck process) to simulate simple/non parametrie stochastic differential equation. Thus, our method do not require any additional learning parameters.
We provide important theoretical results. First, the controllable upper-bound E with the inverse exponential ratio induced, which indicates that our method can efficiently con- trol the diverging effect of Wasserstein distance. Second, the concentration inequality of transported uncertain measure is presented, which clearly renders the probabilistic relation between and &
2 RELATED WORK
Curriculum Learning Small-loss Criterion. To handle noisy labels, Han et al. (2088 Yu et al.
(2019); Jiang et al. (2018): Wei et al. (2020): Lyu & Tsang (2020a): Han et al. (2020) adopted curriculum learning or sample selection frameworks. However, these methods only consider a small number of selected samples, where large portion of samples are excluded at the end of the training. This inevitably leads to poor generalization capabilities. However, this conflicts with sample selection methods because large portion of training samples are gradually eliminated. By contrast, our method can extract useful information from unselected samples X ~ (i.e., uncertain samples) and enhance these samples (e.g., X' F F1) for more accurate classification. Chen al. (2019) iteratively apply cross-validee cross-validation to randomly partitioned noisy labeled data to identify most samples that have correct labels. To generate such partitions, they adopt small-loss criterion for selecting samples.
Loss Correction & Label Correction. Patrini et al. 201 Hendrycks et al. (2088;; Ren et al. (0018) either explicitly or implicitly transformed noisy labels into clean labels by correcting classin- cation losses. Unlike these methods, our method transforms the holistic information from uncertain samples into certain samples, which implicitly reduces the effects of potentially noisy labels. While correction of label noisy by modifying the loss-dynamics do not perform well under extreme noise environments, Arazo et al. (2019) adopt label augmentation method called MixUp Zhang et al. (2018).
Due to the technical difficulties, we define our central objects pre-softmax space rather than label space, Lee. the space of (()),a(Y), where indicates softmax function. Please refer Appendix for more details.
2

---

Under review as a conference paper at ICLR 2021
Distillation. Li et al. (0119) updated mean teacher parameters by calculating the exponential moving average of student parameters to mitigate the impact of gradients induced by noisy labels. Lukasik et al. (2020) deeply investigated the effects of label smearing for noisy labels and linked label smoothing to loss correction in a distillation framework. Similar to these methods, our method leverages the useful properties of distillation models. We set as a pivot measure, which guides our normalization functional Fu for uncertain measures. This is similar to self-distillation because uncertain training samples are forced to be normalized to those of past states.
Other methods. Lee et al. (2019) induced a robust generative classifier based on pre-trained deep models. Similar to our method, Damodaran et al.] (2019) designed a constraint on the Wasserstein space and adopted an adversarial framework for classification models of noisy labeled data by implementing semantic Wasserstein distance. Pleiss et al. (020)) identify noisy labeled samples by considering AUM statistics which exploits differences in training dynamics of clean and mislabeled samples. In most recent work, [Li al. (2019a) adopts semi- comi-supervised learning (SSL) methods to deal with noisy labels where the student network utilizes both labeled/unlabeled samples to perform semi-supervised learning guided by the other teacher network.
3 DISTRIBUTIONAL NORMALIZATION
Because our main target object is probability measure (distribution), we first define an objective function in distributional sense. Let i be cross entropy and f be a corrupted label random vector for an unknown label transition matrix from clean label r which is independent of X, with label transition matrix Q. Then, a conventional objective function for classification with noisy labels can be defined as follows:
min Tul = min Ex=m.flQ [(X:;) f)]
(2)
However, due to the significant changes in label information, the conventional objective function defined in equation 2 cannot be used for accurate classification. Instead of directly using uncertain samples X ~ μ as in previous works, we normalize μ in the form of a metric ball and present a holistic constraint. For a clear mathematical description, we first introduce the following definition.
Definition 1. (Wasserstein ambiguity set) Let P2(R)) {u Euddff (Zo, x) < oc, Vxp {Rd] be 2- Wasserstein space, where d denotes the number of classes, dg is Euclidean distance defined on Rd. Then, we define a Wasserstein ambiguity set (i.e., metric ball) in this space as follows:
(v, - { E P2 (R.) Walu, v) S {)),
where W2 denotes the 2- Wasseerstei distance and v is the pivot measure.
Then, we propose a new objective function by imposing geometric constraints on μ as follows:
min FuEBw (M,)),
J [Fp] + J [<] = min Exiinxxx.oo ((X:; f)] + Ex=ça.r [))
(4)
#
where F: P2(4) + (2R is functional for probability measures, which assures the constraint
on Fu Fu E By, ()) and our main objective. The right- side of equation equation
is equivalent vectorial form of distributional form in left-hand side. While our main objects are defined on poe-sof,max, both probability measures μ4 and & is parameterized by neural network with parameters 0. This newly proposed objective function uses the geometrically enhanced version of an uncertain measure Fu with a certain measure ç. In equation 4 probability measure v is defined as follows: v arg min I [Ex]]. where & denotes a certain measure the current k-th iteration and k* E 11-11 = [1, [1..... k- 1}. In other words, our method finds the best probability measure that represents all certain samples so far at training time, where the uncertain measures are transported to be lying in the Wasserstein ball centered on v. In equation 4 the Wasserstein constraint on Fu enforces uncertain measures statistically resemble v from a geometric perspective (i.e., W2(,, Fu) S E).
Now, an important question naturally stems from the aforementioned analysis: how can we select the optimal radius e? Clearly, finding an F that induces a small & ~ 0 is suboptimal because Fu R v and using objective function J[Fuu ss v/ can lead to the following critical problem. As the optimization process proceeds, enhanced uncertain samples X' ~ Fu contribute less and less, because it is statistically identical to v. meaning our objective in equation 4 would receive little benefits from these transported uncertain samples. By contrast, if we adopt large radius for e, enhanced uncertain samples will be statistically and geometrically unrelated to v, which causes the normalized measure Fu to yield large losses and violates objective.
3

---

Under review as a conference paper at ICLR 2021
Bwg (U,E)
Test accuracy
73-
V
Flud
E
d1
N.
WDN
d2
(2(R
Cross-entropy
WDN
Venilii
30
(a) Divergence or convergence from V
(b) Classification accuracy
Figure I: Accuracy begins to drop when the uncertain measure begins to diverge from v. In classification models with vanilla cross entropy losses, the uncertain measure μu can easily diverge from v in the Wasserstein space (the red dotted line in (a)), which induces an accuracy drop (the red dotted line in (b)). By contrast, the proposed WDN can prevent such divergence by normalizing je onto Wasserstein ambiguity set Bw, (v,E) E) (the black dotted line in (a)) and can consistently enhance accuracy as iteration proceeds (the black line in (b)). Please note that di and d2 denote the first and second terms in equation 5 respectively, and d1 + d2.
To overcome two problems above and select the radius, we make a detour, i.e., a Gaussian measure, for cutting the path between and Fp (i.e., v N(m.. En) - Fp) rather than directly calculating the geodesic between v and Fu (i.e., v - Fu). Specifically, we decompose the original constraint in equation into two terms using the triangle inequality of the Wasserstein distance:
W2 (V, Fu) S
W22 W2 (v,. Ev)) + W2 (W(m,;
(5)
dy: statistics
dy:
Normalization
The first intrinsic statistics term sets a detour point as a Gaussian measure, for which the mean and covariance the same those for (i.e., m, [Y] and E, Covy (Y)) The Wasserstein upper bound of this term is only dependent on the statistical structure of y because (my, 2..) dependent on и. Thus, this term induces a data-dependent, non-zero constant upper bound whenever * N and can prevent the upper-bound from collapsing to E - O, regardless of F. This gives huge advantage when dealing with E because the first term can be considered fixed constant during the training. The second normalization term represents our central objective. F facilitates geometric manipulation in the Wasserstein space and prevent uncertain measure μa from diverging, where is normalized onto the Wasserstein ambiguity Bw, (V, e) in Fig The theoretical/numerrces advantages of setting detour measure as Gaussian is well-explained following section.
3.1 WASSERSTEIN NORMALIZATION
In the previous section, we present a novel objective function that imposes a geometric constraint on such that the transformed measure Fu lies in Bw, (v, E) for v. Now, specify F and relate it to the Gaussian measure (generally Gibbs measure). For simplicity, we denote No, = Nm(m,, Ep).
Proposition F R x P2 - P2 is a fanctional on the probability measure such that F [1, = pt. where du p,dNe, dN,, dq,dx, and pt is a solution to the following continuity equations:
0.p V (HoVr),
(6)
which is read as 0,p(t, x) V (p(t, log q(t, x)) in distributional sense. Then, uniquely defined functional F([-] F(t. normalizes onto By, (Nu, K2 (ji),, where K2(p) > 0 is constant that depends on
It is well known that the solution to induees geodesic in the 2- 2-Wasserstein space (Villani (2008)), which is the shortest path from μ (4== to Ns, The functional J. generates path for prs which the distance is exponentially decayed according to the auxiliary variable and constant K2, meaning (N < Kye- This theoretical results indicates that the Wasserstein distance of second term in equation can be redured/controlled with exponential ratio. Thus, by setting different t, our method can efficiently control the diverging distance in equation5]. Unfortunately. typically intractable to compute the partial differential equation (PDE) in equation
4

---

Under review as a conference paper at ICLR 2021
Algorithm 1 Wasserstein Distributional Normalization
Require: a [0, 0.2). e E [0.1, 0.65], = 64, A. = 10-1, - 0.001, for = to K (i.e., the total number of training iterations) do
1) Select uncertain (1 p).N and certain pN samples from the mini-batch N.
{ ) (nSaN) ~ Ek, (Xx) (nS(1-p)N) pi
2) Update the most certain measure V.
if J [u] < [[v] then
f- Ek, m, t- E [Yx]. and Σ. f- Cov [Yr]
end if
3) Update the moving geodesic average ((mm", , Solve the Ricatti equation TE,, =EEw-
E" =(1 ((1-00 L aT) Ex ((1 La + aT) and m" =(1 -1) omgg
4) Simulate discrete SDE for 7 steps.
for t= to T 1 do
X
Vo(XE,
+ E2 s.l. {x ue, ( (Kkx=T) ~ Fipe
end for
5) Update the network with the objective function.
J [Fuu]= + J [Ex] = Byr ((Ka, f)] Eg, [(X)FF)
end for
To solve this problem, we adopt particle-based stochastic dynamics, which enables tractable com- putation. There exists a unique iterative form corresponding PDE in equation 6 which is called as multi-dimensional Ornstein- Ulenbeck process, which can be approximated using particle-based dynamics. In particular, we draw N1 -) uncertain samples from single batch of N samples using equation I for hyper-paranenn parameter 0 S e S 1. We then simulate a discrete stochastic diffcrential equation (SDE) for each particle using the Euler-Maruyama scheme as follows:
X = X (X?"; m,) A + V 2 AAZZ"
m.). ( (1.... NN1 ,)}. dg is Euclidean distance, and N is single mini-batch size. We selected OU process as our stochastic dynamic due to the following reasons: First, we want to build computationally efficient, and non-parametrie method estimate/minimize the second term of equation 5 The SDE in equation 7 corresponding OU process have simple form with fixed drift and diffusion terms which is invariant over times which makes us induce the non-pan panaaaraettrc representations of simulation of SDE. While the simulation of equation Z is just non-parametric for-loops in implementation algorithm, our method is computationally very efficient compared to other baseline methods such as Han et al. (2018). Second, when estimating empirical upper-bound of Wasserstein distance, OU process allows us to use explicit form called Meheler's 's formula which can be efficiently estimated (Please refer to Appendix for more details). The overall procedure for our method is summarized in Algorithm [
where o (x); 7d
m,)
3.2 WASSERSTEIN MOVING GEODESIC AVERAGE
In our experiments, we observe that the best measure v is not updated for a few epochs after the training begins. This problematic because y diverges significantly from the current certain measure Et. which is equivalent to the normalized measure Fu diverging from Éx. meaning Xr and Y become increasingly statistically inconsistent. To alleviate this statistical distortion, we modify detour measure from Nr to other Gaussian measure, which allows us to capture the statistics of both & and v. Inspired by the moving average of Gaussian parameters in batch normalization Toffe & Szegedy (2015), we propose the Wasserstein moving geodesic average. Specifically, we replace Gaussian parameters {m., } with {mX, Ea such that m = (1 a )), + am,, and " ((1 11-a2 L + aT) Σ, ((1 I)Ia + aT), where T is solution to the Riccati equation TE, ΣEe- Therefore our final detour Gaussian measure set to No == (m)())) E(a)), < a <
4 THEORETICAL ANALYSIS
In 5 we select the detour point Gaussian measure because this measure can provide statistical structure, which is similar to that of the optimal v.. In addition to this heuristic motivation, setting detour point as a Gaussian measure (Gibbs measure) also provides theoretical advantages, e.g., the theoretical upper bound the Wasserstein constraint terms. this section, we investigate the explicit upper bounds of two terms in equation 5 which are naturally induced by the SDE.
Pleasee refer Appendix C.4 for more details.
5

---

Under review as a conference paper at ICLR 2021
Proposition 2. A scalar 0 3 < exists and depends on resulting in the the following inequality: W2(2, S Ki(e) V (ee + K2(v)],
(8)
where Amaax (Fez) denotes the maximum eigenvalue of the covariance matrix Ex, and for some constant K1 0C, have Ki(v) v /d3Amam (2x) + | 1l2 which only dependent on v.
Intuitively, (2( be interpreted
indicator that tells us how the uncertain measure μ is diffused, whereas the designed term e e "2(()) controls the upper bound of the Wasserstein distance using a variable L. The other term K2(v) does vanish even with very large I, which assures non-collapsing upper-bound s.
Proposition 3. (Concentration inequality for the normalized uncertain measure). Assume that 00), there are some constants T ,0), 0 such that the following inequality holds:
EFralf - S (L (1+T))=./AAV/
f f (R3).
for E Sym and D(A, Eny) S. aq for some > with any metric D defined on Sym In this case, there is 6 such that the following probability inequality for uncertain measure is induced:
V20 Kate)
Fru -onll- 2 8) S 6e
(10)
where a denotes a soft-max function.
In equation [0] we show that the label information induced by the normalized uncertain measure is close to that of most certain measure [o). where the upper bound is exponentially relative the initial diffuseness of ja (i.e., K2(n)). Because the upper bound of the probability inequality does not collapse zero and Fru is concentrated around the most certain labels (i.e., (e,,,EE(r)) [o))) ). the uncertain sample XT ~ Fr helps our method avoid parppatttrrizatoon.
4.1 EMPIRICAL. UNDERSTANDINGS
We investigate the theoretical upper bound of the Wasserstein ambiguity (i.e., radius of the Wasser- stein ball) for Fu and its corresponding probability inequality. To provide more in-depth insights into the proposed method, approximate the upper bound and demonstrate that our Wasserstein normalization actually makes neural networks more robust to label noise.
As verified previously, according Proposition 2 the following inequality holds:
W3 (F.,,) S = Ki()) V (Ka(v) + K2(F.1)).
(11)
Because the first term K1(v) (v) is constant, dependent on and generally small compared to the second term with / S T, we only examine the behavior of the second term K2(v) + K2(Fe1s), which can be efficiently approximated using simple form. Because detour measure is Gaussian, we have the following inequality for any h E Co (Rdd
K2(p)
lim | Ex.z~Mi ( a-0 s
V1-e Z (m))+ -
h(x) S K2(H),
(12)
where this equality holds if h is selected to induce a supremum over the set Cg For approximation, we simply consider h (XX) ||x||2 test function. In this case, the following inequality naturally
holds: (2(v) + K2 (F)) < K2(v) + (2(Fu) < K₁ (v) (K2(v) + K2(Fp)) = E. Thus, can be considered as approximation of the theoretical upper bound & suggested in Proposition 2
Subsequently, we investigate the effects of Wasserstein normalization based on K2(u) in equation 12
(1) The proposed WDN ensures that the Wasserstein ambiguity bounded. We examine the relation between é and test accuracy in an experiment using the CIFAR-10 dataset with symmetric noise at a ratio of 0.5. Fig presents the landscape for the -scaled cumulative average of é and test accuracy over epochs. The red dotted lines represent the landscape of the vanilla network with cross-entropy loss, where & K2(v))+ (FF=oo)) and le is the epoch index. In this case, the time constant is set to zero, because Wasserstein normalization is not employed for the vanilla network. The black lines indicate the landscape of the proposed method, where éN = K2(Px) + (2F
Please refer to Appendix C.2 for additional details.

---

Under review as a conference paper at ICLR 2021
Upper bound
Test accuracy
Vanilla
70
60
50
40
WDN
Vanilla
10
15
20
25
45
65
2.3
WON 10
WDN, 20
22
60
VIDN
WDN_10
WDN
WDN_20
2.
65
Epochs
Epochs
Figure 2: Relation between the approximated upper bound È and test accuracy.
in this case. is noteworthy that the test accuracy of the vanilla network begins decrease after 13- epochs (red-dotted vertical lines in the top-right plot), whereas the Wasserstein ambiguity (i.e., upper bound of the Wasserstein distance) increases quadratically in the top-left plot. These experimental results verify that the distance between uncertain and most certain measure (i.e., v/ becomes large in the 2- W-Wasserstein space without any constraints vanilla networks. They also indicate a definite relationship between Wasserstein ambiguity and test accuracy. In the proposed WDN, Wasserstein ambiguity can be efficiently bounded (i.e., lim sup, & Rs 2.15) as the test accuracy continues to increase, even after 13-epochs. For detailed analysis, we compute the deviation of empirical upper bound as follows: ÅS εμ ék-1- 1. In the gray regions, the deviation for the vanilla network is grater than 2.5 x 10-2, i.e., A, > 2.5 x 10-2 Then, its test accuracy begins to drop, as shown in In contrast the vanilla network, the maximum deviation of the proposed WDN is bounded above by very small value (sup, Å
Fig 2
10-3)
(2) The proposed WDN helps networks to escape from vree-paramtttiiitioon. To analyze the behavior of deep neural networks under over-paraman parameterizzinnn with and without the proposed WDN. we design several variants of the WDN, which begin at delayed epochs. The green, orange, and landscapes, when our WDN is applied after ka E (10, 15, 20) epochs, respectively. In this experiment, the upper bound é is defined as
blue curves in the second row of represent
R2(x)) + R2(Fr=o/ke), if < ka, Ex (2(v)) + (2(Fi=TRa), else k 2 ka-
(13)
Consider ha 20, which is represented by the blue dotted vertical lines. Before our WDN is applied (i.e., k kg), the network suffers from over- poeepparamtterizaionn, which induces a significant performance drop, as indicated by the blue curve in the bottom-right plot. However, the network rapidly recovers to normal accuracy following Wasserstein normalization (i.e., k 2 k2). Please note that similar behavior be observed in the green and orange curves. In particular, the orange curve produces less fluctuations than the blue curve in terms of test accuracy. This indicates that the proposed WDN can help network escape from over- parametrrization by imposing geometric constraints on the Wasserstein space with proposed method.
(3) The proposed WDN can derive data-dependent bounds according to different noise levels. Another interesting point in Fig is that all curves, excluding the red curve, converge to specific numbers 2.15 & = lim inf; Ek I lim & == 2.2. The upper bound is neither overly enlarged nor collapsed to zero, while the lower bound g is fixed for all curves. We argue that this behavior stems from the geometric characteristics of the proposed method, where the first term in equation namely W2(,,N,) x K2(v), is a non-zero data-dependent term that minimized by the proposed geometric constraint. Therefore, can derive the following relationship:
[W2(v, Fu) S (2(,,N,) + W2 FF) 4 [K2(v) - (2( El 4
(14)
This empirical observation verifies that a detour point, which is set as a Gaussian measure, can induce the dete-dependnnt bound (g, E), where our data-dependent bound can vary according to different
7

---

Under review as a conference paper at ICLR 2021
Table I: Average test accuracy (%) on the CIFAR-10/100 dataset over the last 10 epochs with various noise corruptions. The symbol indicates scores provided by the corresponding authors. cDN denotes our WDN combined with a co-teaching network. The best results are boldfaced.
Methods
Symmetric 20%
Symmetric 50%
Asymmetric 45%
Vanilla
71.91 + 43/44.444 + 336
+ .27
49.06 + 1.02/31.85 + .85
MentorNet*
80.76 + .6/52.66//2111 13 .40
71.10 + 1.00
58.14 + 38/31.60 + .51
GCE
84.68 + .0555..6/6666 +
0L.80.111/37.00 .08
61.09 + 11/.118/3311111 1.14
RoG*
84.32 / 58.16
76.67 / 45.42
71.26 / 43.18
JoCoR
85.73 = 19 /53.01 + .04
79.41 = 25 /43.99
64.21 + 11/21.12221151 .22
NPCL*
84.30 07/50.7/5555111 .09
77.66 + 00040...942...0000
SIGUA*
S 84
< 78
< 65
DivideMix
81.13 + 18 49.41 + 25
68.93 = 33 34.24 .63
WDN
87.40 23 55.111 .29
82.89 13/48.45 27
76.12 29/38.233322 331
Co-teaching
78.23 - 277 .09
72.81 20/3419 - 50
70.46 .5/34.885.5553 .12
Co-teaching
80.64 =: 15.56..1516..1111 .09
58.43 30/37.88888888811 = .06
70.78 111332.8 .5
cDN
87.12 + 11/11.6/77777 33
76.06 = 28/42.33 38 = 28 74.11 35. .37
Table 2: Test accuracy on the CIFAR-10 dataset with open-set noisy labels from CIFAR-100.
Methods
Vanilla
GCE
Co-teaching
Co-teaching"
JoCoR
WDN
Accuracy
38.12
46.57
35.77
42.57
47.73
51.28
noise levels and efficiently leverage data-dependent statistics.
Fig indicates that classification
models with more stable data-dependent bounds also induce more stable convergence in test accuracy.
5 EXPERIMENTS
5.1 EXPERIMENTS ON THE CIFAR-10/100 DATASET
We used settings similar to those proposed by Laine & Aila (2016); Han et al. (2018) for our experiments on the CIFARIO// dataset. We used 9-layered CNN baseline architecture with batch size of 128. We used the Adam optimizer with (31, B2) = (0.9, 0.99), where the learning rate linearly decreased from 10-3 10-5
Synthetic Noise. We injected label noise into clean datasets using a noise transition matrix Qi, Pr(f = jir = i), where a noisy label i obtained from a true clean label r. We defined Qi, by following the approach discussed by Han al. (208). For symmetric noise, we used the polynomial, 1 + 1.78r + 0.04 for 0.2 < S 0.65, where r is the noise ratio. For the asymmetric noise, we set @ to 0.35. To select the enhanced detour measure, we set to 0.2 for the Wasserstein moving geodesic average in all experiments. We trained our classification model over 500 epochs because the test accuracy of our method continued increasing, whereas those of the other methods did not. We compared our method with other state-of-the-art methods, including [Metorrett, Jiang et al. (2018)) Coo-eaching. Han al. (2018)). [Co-teaching+, Yu et al. (2019)) |GCE, Zhang & Sabuncu 2018) RoG. Lee et al. (2199) |. JoCoR, Wei et al. (2020), INPCL, & Tsang (2020b) [SIGUA, Han et al. (2020)), and Divideeix, Li al. (2019a)). As shown in Table [ the proposed WDN significantly outperformed other baseline methods. Please note that our WDN utilizes simple Gaussian measure as target pivot measure. Thus, there are potential risks when handling highly concentrated and non-smooth types of noise (e.g., asymmetric noise). Nevertheless, the proposed WDN still produced accurate results, even with asymmetric noise. In this case, a variant of our WDN (i.e., WDN exhibited the best performance.
Open-set Noise. In this experiment, we considered the open-set noisy scenario suggested by Wang et al. 2018), where a large number of training images were sampled from other CIFAR-100 dataset; however, these images were still labeled according to the classes in the CIFAR- 10 dataset. We used a 9-layered CNN, which also used in our previous experiment. For hhper-yaramenne 0.5 and 0.2, respectively. As shown in Table our method achieved state-of-the-art accuracy.
we set @ and a to
Collaboration with Other Methods. Because our core methodology based on small loss criteria, our method can collaborate with co-teaching methods. In Han et al. (2018), only certain samples (Y ~ E) were used for updating colleague networks, where the number of uncertain samples gradually decreased until it reached a predetermined portion. To enhance potentially bad statistics for co-teaching, we taught dual networks by considering a set of samples (Y,Xr), where Xy ~ Frj uncertain samples enhanced using equation
8

---

Under review as a conference paper at ICLR 2021
Aoymmeencc
Oen
Dems
Cortaciing
(o-t0cting
Spertic
Figure 3: Test accuracy for the proposed collaboration model with co-teaching.
Table shows the test accuracy results for the proposed collaboration model with co-teaching network (WDNcot). This collaboration model achieved the most accurate performance for the CIFAR- 100 dataset with asymmetric noise, which verifies that our WDN be integrated into existing methods to improve their performance significantly, particularly when the density of pre-logits is highly-concentrated. Fig reveals that co-teaching quickly falls into prer-parrmettriization and induces drastic drop in accuracy after the 15th-epoch. WDN also exhibits slight accuracy drop. However, it surpassed the baseline co-teaching method by a large margin (+7%) during training. This demonstrates that our enhanced samples XT can alleviate the over parameterizationn issues faced by conventional co-teaching models, which helps improve their accuracy significantly.
5.2 EXPERIMENTS ON A REAL-WORLD DATASET
To evaluate our method on real-world datasets, we employed the Clothing M dataset presented by Xiao et al. (2015), which consists of 1M noisy, labeled, and large-scale cloth images with 14 classes collected from shopping websites. contains 50K, 10K, and 14K clean images for training, testing, and validation, respectively. We only used a noisy set for training; for testing, we used clean set. We set a = 0.2 and o = 0.1. For fair comparison, we followed the settings suggested in previous works. We used pre-trained ResNet50 for a baseline architecture with batch size of 48. For the rre-processing steps, we applied a random center crop, random flipping, and normalization to 224 224 pixels. We adopted the Adam optimizer with learning rate starting 10- that linearly decayed to 5 x 10-6 at 24K iterations. Regarding the baseline methods, we compared the proposed method |GCE, Zhang Sabuncu (2018)) [D2L, Ma et al. (2018)) FWW Patrini al. (2017b) WAR, Damodaran et al. (2019).. SS. Wang et al. (2019). JOFL Tanaka et al. (2018). DMI, Xu al. (2019),, PENCIL, & Wu (2019) and MLNNT, Li et al. (2019b)] Table reveals that our method achieved competitive performance as comparison with other baseline methods.
Table 3: Test accuracy (mean, %) on the Clothing 1M dataset.
Methods GCE D2L
FW
JoCoR WAR SL JOFL DMI MINT PENCIL WDN DivideMix
Accuracy| 69.0
69.47 69.84 70.30 70.66 71.02 72.23 72.46 73.47 73.49 74.75
74.76
5.3 COMPUTATIONAL COST
Because Co-teaching, JoCoR, and DivideMix use additional networks, the number of network parameters is twice (8.86M) as many as that of the Vanilla network (4.43M). In Table we compare the average training time for first 5-epochs over various baseline methods under symmetric noise on the CIFAR-10 10 dataset. While non- parametric methods such as GCE and WDN require less than 12% additional time, other methods that require additional networks spent more time than non-parametric methods. The averaging time can vary according to different experimental environments. In table 4 we measure the time using publicly available code provided by authors.
Table 4: Average training time for the 5-epochs (sec) the CIFAR-10 dataset.
Methods
Vanilla
GCE
WDN
Co-teaching
JoCoR
DivideMix
Time
11.43 + .05 11.53 + .06 12.72 .08 15.88 + ,11 17.88 + .11 34.41 + .53
+9%
+11.3%
+38.9%
+56.3%
+201%
CONCLUSION
We proposed a novel method called WDN for accurate classification of noisy labels. The proposed method normalizes uncertain measures to data-dependent Gaussian measures by imposing geometric constraints in the 2-Wasserstein space. We simulated discrete SDE using the Euler- Marle--ar scheme, which makes our method fast, computationally efficient, and non-parametric. In theoretical analysis, we derived the explicit upper-bound of the proposed Wasserstein normalization and experimentally demonstrated strong relationship between this upper-bound and the over-parannn pree-aaattteizatioon We conducted experiments both on the CIFAR- 10/100 and Clothing IM 1M datasets. The results demonstrated that the proposed WDN significantly outperforms other state-of-the-arr methods.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Luigi Ambrosio, Nicola Gigli, Giuseppe Savaré, et al. Bakry-émery curvature-dimension condition and riemannian ricci curvature bounds. The Annals of Probability, 43(1):339-404, 2015.
Eric Arazo, Diego Ortego, Paul Albert, Noel O' O'Connor, and Kevin Mcguinness. Unsupervised label noise modeling and loss correction. In International Conference on Machine Learning, pp. 312-321, 2019.
Dominique Bakry, Ivan Gentil, and Michel Ledoux. Analysis and geometry of Markov diffusion operators, volume 348. Springer Science & Business Media, 2013.
François Bolley and Ivan Gentil. Phi-entropy inequalities for diffusion semigroups. Journal de mathématique pures et appliquées, 93(5):449-473, 2010.
Pengfei Chen, Benben Liao, Guangyong Chen, and Shengyu Zhang. Understanding and utilizing deep neural networks trained with noisy labels. arXiv preprint arXiv: 005040 2019.
Bharath Bhushan Damodaran, Kilian Fatras, Sylvain Lobry, Rémi Flamary, Devis Tuia, and Nicolas Courty. Wasserstein adversarial regularization (war) on label noise. arXiv preprint arXiv: 03936, 2019.
Bo Han, Quamming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor W. Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. In NeurIPS, 2018.
Bo Han, Gang Niu, Xingrui Yu, Quanming Yao, Miao Xu, Ivor W Tsang, and Masashi Sugiyama. Sigua: Forgetting may make learning with noisy labels more robust. In ICML, 2020.
D. Hendrycks, M. Mazeika, D. Wilson, and K. Gimpel. Using trusted data train deep networks on labels corrupted by severe noise. In NeurlPS, 2018.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In ICML, 2015.
L. Jiang, Z. Zhou, T. Leung. L.-J. Li, and L. Fei-Fei. Mentornet: Regularizing very deep neural networks corrupted labels. In ICML, 2018.
Oliver Johnson and Yurii Suhov. Entropy and random vectors. Journal of Statistical Physics, 104: 145-165, 01 2001, doi: 10. (.23A::111033522::
Samuli Laine and Timo Aila. Temporal ensembling for semi-supervised learning. arXiv preprint arXiv:1610.02222..10 2016.
Kimin Lee, Sukmin Yun, Kibok Lee, Honglak Lee, Bo Li, and Jinwoo Shin. Robust inference via generative classifiers for handling noisy labels. In ICML, 2019.
Junnan Li, Richard Socher, and Steven CH Hoi. Dividemix: Learning with noisy labels semi-
supervised learning. In International Conference on Learning Representations, 2019a.
Junnan Li, Yongkang Wong, Qi Zhao, and Mohan Kankanhalli. Learning to learn from noisy labeled data. In CVPR, 2019b.
Michal Lukasik, Srinadh Bhojanapalli, Aditya Krishna Menon, and Sanjiv Kumar. Does label smoothing mitigate label noise? arXiv preprint trXiv:2003.02819 2020.
Yueming Lyu and Ivor W. Tsang. Curriculum loss: Robust learning and generalization against label corruption. In ICLR, 2020a.
Yueming Lyu and Ivor W. Tsang. Curriculum loss: Robust learning and generalization against label corruption. In ICLR, 2020b.
Xingjun Ma, Yisen Wang, Michael E Houle, Shuo Zhou, Sarah M Erfani, Shu-Tao Xia, Sudanthi Wijewickrema, and James Bailey. Dimensionality-drvee learning with noisy labels. arXiv preprint arXiv: (0M:::80600000 026/2, 2018.
10

---

Under review as a conference paper at ICLR 2021
Luigi Malagö, Luigi Montrucchio, and Giovanni Pistone. Wasserstein riemannian geometry of gaussian densities. Information Geometry, 1(2):137-19,, 2018.
Robert J McCann. A convexity principle for interacting gases. Advances in mathematics, 128(1): 153-179, 1997.
Ivan Nourdin, Giovanni Peccati, and Yvik Swan. Entropy and the fourth moment phenomenon. Journal of Functional Analysis, (1665):3100-2207 2014.
Felix Otto and Cédric Villani. Generalization of an inequality by talagrand and links with the logarithmic sobolev inequality. Journal of Functional Analysis, 173(2)361-400,, 2000.
G. Patrini, A. Rozza, A. K. Menon, R. Nock, and L. Qu. Making deep neural networks robust to label noise: loss correction approach. In CVPR, 2017a.
Giorgio Patrini, Alessandro Rozza, Aditya Krishna Menon, Richard Nock, and Lizhen Qu. Making deep neural networks robust to label noise: A loss correction approach. In CVPR, 2017b.
Geoff Pleiss, Tianyi Zhang, Ethan R Elenberg, and Kilian Q Weinberger. Identifying mislabeled data using the area under the margin ranking. arXiv preprint arXiv:2001.10528, 2020.
M. Ren, W. Zeng, B. Yang, and R. Urtasun. Learning to reweight examples for robust deep learning. In ICML, 2018.
Daiki Tanaka, Daiki Ikami, Toshihiko Yamasaki, and Kiyoharu Aizawa. Joint optimization framework for learning with noisy labels. In CVPR, 2018.
C. Villani. Optimal Transport: Old and New. Grundlehren der mathematischen Wissenschaften. Springer Berlin Heidelberg, 2008 ISBN 9783540710509.
Yisen Wang, Weiyang Liu, Xingjun Ma, James Bailey, Hongyuan Zha, Le Song, and Shu-Tao Xia. Iterative learning with open-set noisy labels. In CVPR, 2018.
Yisen Wang, Xingjun Ma. Zaiyi Chen, Yuan Luo, Jinfeng Yi. and James Bailey, Symmetric cross entropy for robust learning with noisy labels. In ICCV, 2019.
Hongxin Wei, Lei Feng, Xiangyu Chen, and Bo An. Combating noisy labels by agreement: A joint training method with co-regularization. In CVPR, 2020.
Tong Xiao, Tian Xia, Yi Yang, Chang Huang, and Xiaogang Wang. Learning from massive noisy labeled data for image classification. In CVPR, 2015.
Yilun Xu, Peng Cao, Yuqing Kong, and Yizhou Wang. L _mi: novel informattion-theortie loss function for training deep nets robust to label noise. In NeurIPS, 2019.
Kun Yi and Jianxin Wu. Probabilistic end-to-end noise correction for learning with noisy labels. In CVPR, 2019.
Xingrui Yu, Bo Han, Jiangchao Yao, Gang Niu, Ivor Tsang, and Masashi Sugiyama. How does disagreement help generalization against label corruption? In ICML, 2019.
Xiyu Yu, Tongliang Liu, Mingming Gong, and Dacheng Tao. Learning with biased complementary labels. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 68-83, 2018.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez- Pazz mixup: Beyond empirical risk minimization. In International Conference on Learning Representations, 2018.
Zhilu Zhang and Mert Sabuncu. Generalized cross entropy loss for training deep neural networks with noisy labels. In NeurIPS, 2018
11

---

Under review as a conference paper at ICLR 2021
A OPEN-SOURCE DATASET
Transition matrix for CIFAR-10/100. For the experiment summarizze in Table I we implemented open-source code generate the noise transition matrix discussed by Han et al. (2018) as well the 9-layered CNN architecture hhttps://githu... com com
teaching)
Open-set noise.
we used the same dataset for the experiment summarized in Table 2
open-set noisy labels presented by [ee et al. (2019) (https: /tl::///ittt //gttttt......gg cotpp//apooopookaoaa/a RoGNoisyLabel)
Clothing For the experiment summarized in Table we used the open-source dataset presented by Xiao et al. (2015) (https://githui com Cacooo//yys/aaeeee wwt.:/yyyyyoooiyyllloeeeee
B COMPARISONS TO RELATED WORKS
Methodology
Parametric Class-dependency Distillation
Sample-weight
Sample-selection
DivideMix
X
X
X
Co-teaching
X
V
XX
X
JoCoR
X
V
X
MLNT
/
V
X
Ren et al. (2018)
X
X
NPCL
X
X
GCE
X
X
/
WDN
X
X
X
X
X
Table indicates that no previous methodologies can conceptually include our method.
Because the solution to the Fokker-plank equation can be explicitly calculated without any additional parameters, our method is fully non-pate non-parametric (in terms of additional parameters beyond those required by the original neural network). By contrast, co-teaching is parametric because it requires a clone network with additional parameters that are copies those in the original network. Simi- larly, MINT requires an additional teacher network for training, which also contains number of parameters.
Many method based small loss criteria select certain samples, whereas method uses the combination of pN certain and (1 (1-p) normalized uncertain samples. Therefore, our method fully leverage the batches of training datasets, where (1 p)N pN = N. Additionally, method does not assume any class-dependent prior knowledge. Rather than considering class-wise prior knowledge, our method uses holistic information from both certain and uncertain samples (i.e., Y and X in the logit space. Other sett--classbbsee model, such as MLNT, assume class-wise meta prior knowledge from teacher network.
In Arazo et al. (2019), they assumed the beta mixturr model as a label distribution on label space. But due to the non-deterministic type of noisy label distribution, it sometimes fails to train with extremely non-uniform type of noise. For example, Arazo et al. (2099) reported failure case with Clothing IM dataset. seems that fundamental assumption on noise model mixup will be improved future work. Similar to this method, our work have trouble when dealing with synthetic asymmetric noise with high ratio where relatively large performance drop is observed in Table (despite our method produces second best performance in the table).
Most recent work Li al. (2019a), they also adopt Co-train by implementing additional dual network, based on SSL. We predict that the Wasserstein distance between labeled and unlabeled probability measures is well-controlled their method. We think that applying the OT/Markov theory (as in paper) their method will broaden the understanding LNL problem.
but much sophisticated methodology called Co-divide/
In contrast to sample weight methods such GCE and NPCL, which require prior knowledge regarding the cardinality of the training samples to be weighted, our method is free from such assumptions because Wasserstein normalization is applied in batch-wise manner.
12

---

Under review as a conference paper at ICLR 2021
C TECHNICAL DIFFICULTY FOR APPLYING GENERAL OPTIMAL TRANSPORT/MARKOOY THEORY TO LABEL SPACE.
Let X, Y be uncertain and certain samples in pre-softmax feature space. And assume that consider the distributional constraint on label-space (the space of a (X),, (Y), where a denotes the soft-max function). This space is not proper to define the objective function such (5). Because, all the samples in this label space is of the form (XX) = [ai, 02; ... Un] such that Σi=1 04 = 1, thus label-space is d-dimensional simplex Ua which is subset of Euclidean space Ua Rdd this case, the definition of Wasserstein space in equation (4) is unacceptable while dg is not true metric on Ua. The Wasserstein space P2 (Ua) is merely investigated in the mathematical literature which makes unable to use all the technical details and assumptions, theories developed in the P2 which are theoretical ground our work. But, if we look this problem slightly different point of view, for example, consider pre-softmax R. P2 (R") our base space. In this case, all the technical issues/problemss when we try to use OT tools in P2 (Ua) can be overcome/ignored. while softmax is non-parametric one-to-one function connecting pre-softmax feature space TRd to Ua there exists unique labels in Ud as mapped point of the manipulated uncertain samples. Even though our objects defined on pre-softmax space, the theoretical analysis in Proposition 3 contains sofumax function to evaluate the concentration inequality of proposed transformation F affecting in label-space Ud-
D MATHEMATICAL BACKGROUND
In this section, we introduce important definitions, notations, and propositions used in our proofs and the main paper.
D.I NOTATION
We denote f as push-forward of μ through f. (RR) denotes the set of oc-class functions with compact support in Rdd, For the normo of the function J, we denote |I/l = ( du) The Hessian matrix of the function f is denoted as Hess [8, Sym, denotes the space for semi-definite positive symmetric matrices of size d x d. [I Lip denotes the Lipschitz norm of the function f. For any matrix A Ma, we let |lp denote the operator norm of A.
D.2 DINFUSION-IVARIANCE AND HYPER-CONTRACTIVIT
Definition 2. The Markov semigroup (Pi)ızo in Rd acting a function f E C is defined as follows:
Pif(x)= f(z')p((z (z,))')
(15)
where pe(x, dx') is transition kernel that is the probability measure for all 2 0.
Definition 3. (Diffusion Operator) Given Markov semi semi-group Ps at time the diffusion operator (ie., infinitesimal generator) L of P. is defined as
8² dy:
Lg(y) = lim (Pig(u) - -vv)) = Σ 2-+0
BO())(())
Σ (()
g(v). (16)
By,
where B and A are matrix and vector-valued measurable functions, respectively. B denotes the (i, function functiion of B and A' denotes the i-th component function of A.
Definition 4. (Diffusion-invaat
Measure) Given the diffuusion operator C, the probability measure is considered to be invariant measure to L when Ex-rl_f(X) 0 for any f E C
Lemma 1. (Infinitesimal generator for the multivariate Gaussian measure, Bolley & Gentil (010 The Gaussian measure N,, == N(m,, E,) with mean m, and covariance M, is invariant measure according to the following diffasion-operaut L:
Lf(x) = E, Hess([((z)) (a m.) V)(x), Vʃ E CRC),,
(17)
where Bdd (x) [Eglis is constant function, and (1)))= = x² m
13

---

Under review as a conference paper at ICLR 2021
This generator serves main tool for the geometric analysis of the upper bound In Section 4.1
in the main paper, introduced an approximate upper-bound K2(u)) without any general description of the inequality involved. We now introduce the underlying mathematics for equation [2] Because detour measure is Gaussian, there is a unique semi-group P.h called the multidimensional Ornstenn-Ulenbeck semi-group that is invariant to N2. Specifically, Pt is defined as follows:
e-2 (a((X)= = EznNi (s ( V1- e (=2 Z + m
vh E Co
(18)
The invariance property of Pt relative to our detour measure is naturally induced by the following Proposition:
Proposition 4. We define C: Rd + Rd and ((X) = AX + b such that A E Sym E R and select an arbitrary smooth h E (IR.). We then define the diffusion Markov semi- remi-group P.h as follows:
Pah(X)= = EZ-N (e (I-e-= -
(19)
Then, (A2,bb) is invariant with respect to Ps. meaning the following equality holds for every and 2 0:
J - -.
(XX))
(20)
Proof. For simplicity, denote b)
J P.MX))NNCX))
/ H1EE "X (1
(21)
= / J / + 1 2
The second equality holds because C is linear in Rd. Let cos and e 20 sin 0 for any S A < 2m. Then, we define Φ as (0Z', Z) = 7 v1 6-2 cos(() 7' sin Z, and T(Z', 2) Z. Based on the rotation property of the standard Gaussian measure, one can induce the following equality.
(N WON) o ((N W\) o0
(22)
However, we know that dN [X)] = dNc = ((2)) (227 - -) By combining equation 21 and equation 22
can derive the following result:
J ho ho C(e + V1 - 2dd @ =
(X)
f M(X]dN (
MX))NTT
= / M(X)NNo(X)
(23)
Proposition demonstrates the invariance property of the defined semi-group, If we set A E3 b m,, then we can recover equation [8
We are now ready define the approximation of K2(p) in terms of semi semi-group group invariance. Specifi- cally, for any vellev smooth h, we define the following inequality:
(P,)(x)) (x(x)))n((x))
(2(p) = Ex [Ch()) = lim Expe
8-0
(24)
e- vi- ( -m) n(x) & Kz(1)).
lim x-0 s
Fx Ex,z~,N
This inequality holds if is selected to induce a supremum over the set C where sup, R2(H, h) sup, [xx)] = K2(p), Although more sophisticated design for the test function h will induce tighter upper bound for Rg, determined that the I-z-norm is generally sufficient.
14

---

Under review as a conference paper at ICLR 2021
Definition 5. (Diffuseness of the probability measure) We define the integral operator K₂ W2(RR) + R as follows:
K2(p) -V
sup SECO Jgd du(x).
(25)
According to Definition 4 we know that f (X(X))N((X) 0 for any f. Based on this observation, is intuitive that K2 estimates how the probability measure y is distorted in terms of diffusion invariance. While this measure takes supremum over the function space C it searches for function that enables the estimation of maximal distortion. Because the value K2 is entirely dependent the structure of μ4, K2 be considered uncertain measure 7l is fixed over one iteration training.
constant for the sake of simplicity if the
Definition 6. (Diffusion carré du champ) Let f.9 E C. (R.). Then, define bilinear form Te in Co (IR.) x Cg (TR.) as
1 Te( - ((T
- - - e
(26)
We also denote T(f) = I). The bilinear form I' can be considered as a generalization of the integration by the parts formula, where f + I(f)du for the invariant measure μ of L. Definition 7. (Currature-Dimensin
condition, Ambrosio et al. (2015)) We can say that the infinitesimal generator f. induces the CD(p, oc) curvalure-dimension condition it satisfies Ti(f) S ( all C8.
Because our diffusion operator generates semi-group with respect the Gibbs measure, the curvature-dimension condition can calculated explicitly. Through simple calculations, the first- order (c 1) diffusion carré du champ can be induced as follows:
Ti(f) (VV))',
(27)
Similarly, the second-order (c 2) diffusion carré du champ is calculated as follows:
1 T2()) = (Ti(()) - 2r, (J,(N))
(28)
Tr (EW [. =Tr (TE + +T((),
for an arbitrary f C (R*). While Tr (EVV is non-negative, we can infer that I3 S Tz-
In this case, the diffusion operator L defined in Lemma I Linduces the CD(p- 1, oo) curvature- dimension condition. For the other diffusion operators, please refer to Bollll & Gentil (010))
Proposition 5. (Decay of Fisher information along a Markov semigroup, Bakry al. (2013).) If
assume the curvature dimensionn condition CD(p, oo), then [2) S E 4pt (u,
The exponential decay of the Fisher information in Proposition is core property of the exponential decay of the Wasserstein distance, which will be used in the proof of Proposition 2
D.3 FOKKER -LANKK EQUATION, SDE
Definition 8. (Over-damped Langevin Dynamics) We have
dX, (X, (1xX)))))dd my) + E2, where & m,); _d (X,, m,). W, denotes Brownian motion, and d denotes Euclidean dis- tance. The particle X, is distributed X, ~ Pt- The probability density lime- p(x, t) with respect to Xoo converges the Gaussian density Xxc = VEv(Z my) Poo(z) = g(x) x
(29)
-d(z.m.)) "..mmm
In classical SDE literature, is stated that supoSesT X, - x < G(Ne)
- where G(T) is
some constant that depends only T and X denotes the true solution of the SDE in equation 29 While the number uncertain samples is greater than No o > 40, our method exhibits acceptable convergence.
15

---

Under review as a conference paper at ICLR 2021
D.4 GAUSSIAN WASSERSTEIN SUBSPACES
It is known that the space of non-degenerate Gaussian measures (i.e., covariance matrices are positive- definite) forms subspace in the 2-Wasserstein space denoted W2.g = Sym Rd Because the 2- 2-Wasserstein space can be considered as a Riemannian manifold equipped with Riemannian metrics Villani (2008), Wg.g can be endowed with a Riemannian structure that also induces the Wasserstein metric (McCann (997)). In the Riemannian sub- manifold of Gaussian measures, the geodesic between two points (0) NA and 7(1) NB defined as follows Malagò al. (2018)
7(a) N₁ . (m)())) E(a)).
(30)
where maa) = (1 - a a)m, + omg and I(a) = [[1 [1- [1-a)] - (m,, Eny and (mg, En) + (m De,,)
where
TEAT= Σg. In Section 3.2 we set (m,, E,,
Regardless of how is updated, the statistical information regarding the current certain measure & is considered in the detour Gaussian measure, which yields much smoother geometrie constraint on J..
E PROOFS
Proposition 6. Let (H,, be a of couplings between and v, and assume that the noisy label is independent of X. For functional [1p]]= = Ex-xl(X;;f) we define D(1, v) as:
D(u.v)= inf
[T[m] Jvlll.
(31)
rEI(u.))
where D P2 P2 + R. Then, D is the metric defined on P2z, which is weaker than the Wasserstein metric, where D(,,v) S aWa(j, v) for a co + ci (1 f) and some constants cu, C1 >> 0.
Proof.
= J [ - [B, -.
Eu [ (logo(X) o - (1 (log oX)) (Z)))]
< E FE [log o(X) - log (2)] + [1 f) log(1 - o(X)) - log(1 o
(32)
< Er llog (X) (Z)) + E(1 r) P llog(1 (X)) log(1 a())
< c '(()) E X Z|+ ci E(1 f)E -
= +
B |X - Z
By taking the infimum the aforementioned inequality with set of couplings 7 v), we obtain the following inequality:
D(v.)) inf ((.))
[J[1] Jb4ll < Elc, Y (1 inf B,| Z (HIM)
1Elo + -1-
(33)
< El + (1- (1-Y)-)))))
which completes the proof.
Proposition 6 follows from the Lipschitzness of the functional J. where D searches for the best coupling to derive the minimal loss difference between two probability measures. This proposition indicates that inf [ - I [Fu]] is bounded by the Wasserstein distance, which justifies our ge- ometric constraint presented in equation 4 It should be noted that the prior assumption regarding noisy labels is essential for Lipschitzness.
Let F ; R+ x Py be a functional on probability measures such that F [1, [4] = Pt. where dju prdNy, dN,, dqıdæ. and let hr be a solution of the continuity equation in the 2- Wasserstein space defined as follows:
Proposition 7.
d,p = V -
(34)
which is represented as 0,p(t,z) functional Fal.1 Flt, is defined unique and normalizes 4 onto Bwz (Nu, e e', (H)), where K2(p) S oo an integral operator in Definition 5 with respect to μ.
(p(t, x) log x)) in distributional sense. Then, the
16

---

Under review as a conference paper at ICLR 2021
Proof. We assume that the probability measure di is absolutely continuous with respect to the detour Gaussian measure (m,, En) Ny, br < Ny. this case, according to the Radon-Nikodym theorem, there is corresponding unique probability density q(t, x) (e(z) E C such that dp = q.N.v.
Lemma 2. (WI-inequality, Otto & Villani 2000 If the stationary state of #4 with respect to P satisfies lim By Prf] = ffror any J EEG ECO. then the following inequality holds:
d
Mall, S VT(aNN.).
(35)
d
By integrating both sides of the inequality in Lemma 2 with respect to t E (0, oc), the following inequality can be obtained:
W2(ue,NN)) - L/x d
Wa(me, & Lo VIlueNs)dd.
(36)
di+
In the aforementioned inequality, we replace the Fisher information with the diffusion generator L as follows:
W2(,,No) & / Vi(unWsjdtt
(37)
/ / log Pig)duadt.
po
The second equality above is derived by leveraging the properties of the bilinear operator I' (Bakry et al. (2013) 3 Villani (2008)) with respect the diffusion operator L, which is defined as follows:
/ [Pid PRgi-'T(PPigdNN))) - [(log Pig)qeaN, =f f [ log Pıg)d 2 0.
(38)
For simplicity, we denote [g| g" for any g E Cg According to Proposition 5 we can relate Fip = to its initial term μ 1==0 as follows:
Jo / p(- 1o log (()(() s
30 V Ve-an / c(-log Pt=oo) (X)du(X)dd
f
/
e22p sup
(tg(Z)gdN
jo
gErrE
(39)
-e- 2t
(tg(X)du(x)
sup
gEC
-211))())
The second inequality is naturally induced, because the proposed objective function is defined to select the maximum elements over the set of functions g Cp and Lg < Ctg. If the integral interval is set to (0, s), then we can induce Wz(H,, Fip) S p e "=((() Our diffusion--pperator induces p == 1, which completes the proof.
Proposition There is a sealar < B < oo dependent on D such that the following inequality holds:
Wa(u, Fin) < [ d3X Ex) + ||E v "e()))))) + K2(v)]
(40)
As a motivation for setting a detour measure to No. we mentioned the natural property of the non-collapsing Wasserstein distance of (V. # O. However, is unclear from geometric perspective exactly how the upper bound (i.e., (v, < ?) can be induced based on the intrinsic statistics term (i.e., di in Fig. l). Specifically, in the situation where the covariance matrices of and N2 are identical, it is difficult to determine theoretical upper bound without additional tools. The first part of this proof focuses resolving this important issue. The second part of the proof naturally induced by Proposition[] [ Please note that the following proposition, parameter for Wasserstein moving average is set to for clarity.
17

---

Under review as a conference paper at ICLR 2021
Proof. Before proceeding with the first part of the proof, define constant as follows:
- By, u2. (Y,)dd (Y,)ds.
= sup 15isd
(41)
If we assume mild condition such that mins.j infis (vss) ) 2 O(vs), then the integral term in is finite and well-defined. This value will directly yield the upper bound of the Kullback-Leible (KI.) divergence of v. First, we introduce the following inequality.
Lemma 3. (de Bruijn's 's identity, Johnson Suhov (200T); Nourdin et al. (2014)) We let Y v, 7. ~ (O,,,) I) denote standard Gaussian random variable, and let define Y, VYY V $ 7 with the score function defined as (x) = V loggpp( Pa (x) with respect to the random variable Y,. Then, the following equality holds:
L1 ( V.
KL En))= Tr
ds.
(42)
From equation earlier.
42 derive the relations between KI--divergence and the constant / defined
1 Tr (E (e.tv,)))))))) ds
Tr (E,Ez (us (Y.)s, ds 1
2s
(43)
1 Aman
(Ya)
ds S Aman (' E 3ds (
The second inequality holds based on the following element property of symmetric positive-definite matrices:
Tr(AB) - ||Allop Tr(B) = Amaz (A)Tr(B), VA, B E Sym
(44)
It should be noted that because the distribution of is compactly supported (i.e., supp(g) is compact), the maximum eigenvalue of the covariance E, is finite. The other relations are induced by the aforementioned definition. Next, we relate the Kl.-divergence and 2-Saserrseein distance naturally.
Definition 9. (Talagrand inequality for Gaussian measures, Otto Villani (0000) For any non- degenerate Gaussian measure N with a mean 0, the following inequality is satisfied:
Wz(1,N) V2KL(v|N),
Yv E P2(R).
(45)
By combining Definition 9 and equation 43 we can derive the following expression:
Wl,v, Ex)) v
E,)) dB3mma (En) D0.
(46)
According to the triangle inequality for the 2- Wassersteii distance, we obtain:
We(,, Ex)) s (2 E,)) + W2(N(m, Ex), Ex))
(47)
In Appendix C.3, we investigated that the geodesic distance between two Gaussian measures having the same covariance is equivalent to the Euclidean distance between two means. Therefore, we can obtain the following equality:
W m 0|1
(48)
|
where (x) X for any vector E supp(g). Now, by adding the inequalities defined earlier, we can obtain
Ws(v,N(mm, Ex)) < J,,Y ddBAmar (Ew).
(49)
where it is easily shown that the upper-bound is only dependent on the statistical structure of v. Specifically, the term ||E,Y|] represents the center of mass for a density of v and / /332maa (E,) is related to the covariance structure of
18

---

Under review as a conference paper at ICLR 2021
By applying Proposition 8 both Fij and v, we can easily recover equation 5 as follows:
W2(2,Fip) S = (2(,, Ex)) + W2(N En), Foje)
- |E, + v Vd (In) < dBXmaz(Z2 + | Bo V ([eT'K2(u) + K2(v)]
A +e K2(())
(50)
The second inequality easily obtained as (a A b) + S a (b c) for any a, b. 2 0, which completes the proof.
O
Proposition 9. (Concentration inequality for uncertain measures). Assume that there are some constants 11 n.' 00).1 2 such that the following inequality is satisfied:
E, E (E, If1I ( (
(51)
for A E Sym D(A,E,)) S an for case, there is 5 such that the following probability inequality for an uncertain measure induced:
O, and for any metric D defined Sym In this
.
F a Ex[o]l 26 S 6c
R2
(52)
where K. denotes the Lipschitz constant of a.
Proof. Before proceeding with the main proof, first prove the existence of s*. The limit of the interval with respect to 7) converges to singleton {oo} as fo. 20). In this case, equation 51 is the same as the Poincaré inequality for a Gaussian measure No. which can be written
lim [F. [R]++[[]] 9
S lim (1 q 7
(53)
=E,,,(EE.
While the Poincaré inequality in equation 53
uniquely defined, we can find at least one value st X,(w) denote the stochastic process with respect to qe (4)
satisfying equation 51 Let (t,, w)
defined in the proof of Proposition 2
Additionally, let Eo Ex wl) Then, can obtain
the following inequality:
0 Ex
EFAH [2]) S K sup (Evg EF,,
c= c=Erlo] EF,,Hoo] = K
sELip,
(54)
kK2(1) 1+
S {nMA(FFHHM)) (
The first inequality is induced by the assumption regarding the k-Lipschitzness of the function and the second inequality is induced by the Kantorovic--Ruiinttinn theorem. The third inequality is natural because (..-.) S Wo(.-) for any 15 a
3c. because equation[51] equivalent the Poincaré inequality for the measure F. μ. it satisfies the Bakry-emery curvature-dimensie condition CD(11 77, oo). Thus, as shown in the proof of Proposition 2 (i.e., equation 39), the last inequality is induced. Additionally, based on the concentration inequality of F.- [Proposition 4.4.2 Bakry et al. (2013)], we can derive the following probability inequality:
Fo la (x (w)) EF. + 8, 3e राककर
(55)
where the Poincaré constant for Fs- is naturally 1 + 77 and ||o|| Lip = k. Next, we will derive the desired form from equation 55 First, we introduce the following inequality.
o (X,,) > EF [o] +
I+ K2
(56)
The last inequality directly induced by equation 54 because- -c 2 1+. K2. While y, K, and K2 are constants with respect to w, the following set inclusion can obtained naturally:
K K2 S2.
S₁ = {w: oXx- (w)) > EF Eo + +0} 2 {w: a (Xxx [olo
1+
(57)
19

---

Under review as a conference paper at ICLR 2021
For the modified version of the original probability inequality, take probability measure F4. for the sets S1, S2, which is defined as
3e भोंर 2 Fs (fw: (Xxx 2 [ + +.]
(58)
2 F (w: (w)) [o + 1+
K2)
The concentration inequality around [o] is obtained by combining the inequalities induced by and -a follows:
F.
U hE(a.-o)
tw (((xx Ex 2
1 K2)
&
(59)
= Fs (w: (ox(Xx [a]] 2 1+1 7
K2} S Ge vT+4K_ -o+00
The inequality in equation 59 is the general form containing the relation between the upper bound of the probability and (1,, Ki K2)- While this form is quite complicated and highly technical, we choose to present all the detailed expressions of equation 59 the main paper. Rather than that, re-write it in much simplified form for clarity. Specifically, by setting x/2/ + 77) = 0.56 and rescaling 8 26, the aforementioned inequality in equation 59 can be converted into the following simpler form:
v
Fs (x(x.(ww) EE 5) S 6c
(60)
Finally, we set a = Softmax, then the Lipschitz constant induced as x = This proof is completed by setting s* T.
20


Reviews:
[{"review_id": "ew0pxf0UC8X", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper propose a computationally efficient Wasserstein distributional normalization algorithm for accurate classification of noisy labels. An explicit upper bound for the Wasserstein-2 distance is derived and such a bound can be used as an estimator to determine if a network is over-parameterized. Empirical results on CIFAR-10/100 and Clothing1M suggest that the propose algorithm outperforms other SOTA approaches. \n\nOverall, this paper is very well-written and easy to follow. The problem is well-motivated, the proofs looks solid, and the claims are well supported by experiments.\n\nI am positive with respect to acceptance of this paper. First, compare to previous works, the proposed method (WDN) is fully non-parametric, sufficiently leverage the batches of training datasets, and do not assume any prior knowledge on class dependency. Hence WDN is flexible and potential has a smaller generalization error (compare to existing method based on sample selection). Second, I would like to emphasize the technical quality of the paper.  The concentration results in this paper are by themselves quite  elegance.", "extra_scores": {}, "extra_text": {}}, {"review_id": "ZY-eWGpfDVT", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper is a contribution that aims at solving the label noise problem. In this setting, the labels are possibly corrupted, this yielding a potentially significant underperformance of a (neural network) classifier when minimizing the empirical risk. This problem is ubiquitous and important in real life scenarii. The paper builds on the idea of small loss criteria, which favors learning on certain samples in the beginning of the learning process, and gradually incorporate uncertain samples along iterations. The paper proposes a novel type of distributional normalization based on Wasserstein distance. It projects uncertain samples on a Wasserstein ball defined wrt. the certain samples. This process is done with a particle based stochastic dynamics, based on a Ornstein-Ulenbeck process. A theoretical Analysis is given, along with results on classical datasets in the symmetric noise setting, open noise and a real world dataset (clothing 1M), for which it achieves very good performances compared to state of the art competing methods. \n\nWhile the overall idea is interesting, and the results described in the paper seem impressive and state-of-the-art on many of the tasks considered, I believe that the paper is badly written and very hard to follow. In the reviewer’s opinion, the major problems arise from:\n - a lack of introduction on the label noise problem, and a more comprehensive overview of the so called small loss criterion and its rationales;\n - despite an apparently rigorous mathematical framework, there are a lot of vague assertions (see minor remarks) and arbitrary choices (two examples: certain and uncertain distributions are looked upon in the pre-softmax logins, why ? Is the continuity equation of proposition 7 the only choice to model \\mathcal{F} ? Some parts are really unclear (Why taking a detour ? what is the role of the Wasserstein moving average ? Is solving the SDE corresponding to the OU process the only solution to perform the normalization ? There are many algorithms in the literature that allows to estimate an OT mapping without solving this SDE)\n \nFinally, whereas symmetric noise is considered in the experiment, I wonder how much the method is amenable to the more realistic setting of asymmetric noise, which is more likely to occur in the real life scenarii.\n \nAs a conclusion, I think there is a lot of interesting ideas and material in this paper, but the writing should be improved and clarified. I am willing to revise my note positively given that those aspects are taken into account in a revision of the paper. \n\n\nMinor comments:\n- why taking the pre-softmax activations to build \\ mu and \\epsilon ?\n- some sentences are rather vague : ‘a functional that can manipulate \\mu to induce geometric properties...’ ‘the pivot measure can be represented as a mini-batch containing the best combination of certain samples’ ‘the other iteration ‘ of which process ?\n- Définition 1: ‘d’ is used for the distance symbol and the dimensions of the input signal\n- Why Eq. 7 is a proposition ? It is rather a modeling choice \n- Typo p8 CIAFAR-10 -> CIFAR-10\n\n### After response to authors\nI thank sincerely the authors for providing a detailed answer to my concerns. I changed my note to 6. I believe the paper is interesting and show strong empirical evidences that the method is worth considering. I am not giving a higher grade because in my opinion the writing of the paper could be signifcantly improved. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "HMX2BWDev2e", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\n\nThe paper introduces a novel objective function by imposing geometric constraints on the logits of uncertain samples. The authors' approach is to map the distribution logits of uncertain samples onto the 2-Wasserstein ball centered on the measure of certain samples. To overcome the dilemma of selecting the ball radius, the authors propose a surrogate objective, namely Wasserstein Normalization. An SDE grad flow is proposed for solving the Wasserstein normalization. The paper also keeps the Gaussian parameters as moving average during training in light of batch normalization. The paper both theoretically and empirically validate their method.\n\nContributions\n\ni) Proposal of a novel Wassersterin Normalization objective for uncertain samples. \n\nii) Theoretically giving a verifiable upper bound of the constrain term.\n\niii) Extensive experiments on synthetic and real-world datasets.\n\nIssues:\n\ni) The motivation of normalizing the uncertain measures into the 2-Wasserstein ball of certain measure is unclear to me. And it seems irrelevant to the practical problem. Yes, I agree that we should prevent the over-parameterized network to overfit the uncertain samples since the majority of them are noisy samples (network tends to fit noisy samples slowly). Based on this observation, the idea of discarding/relabeled uncertain samples is widely adopted in this field. The paper's approach belongs to this category. But the motivation of Wasserstein Normalization for uncertain samples is weak. I cannot directly see its benefits. Imagining the network has high confidence for certain samples (which is generally true in practice), then the mean of gaussian $m$ is uniform categorical distribution.  The paper's approach is just injecting noise onto the network output of uncertain samples.\n\nii) Does the simulation of the SDE makes the training much slower? Since we need the calculate the grad flow for every single sample during every iteration of the inner loop. \n\niii) Could the authors better explain the dilemma of selecting the ball radius $\\epsilon$ in Sec 3? I cannot see why the benefits decrease when $\\epsilon\\approx 0$.\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "3A9WwuZoedA", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nThis paper aims to deal with the learning of noisy/corrupted labels based on the small loss criterion. If I understood well the idea is to consider a new loss function on the Wasserstein space to learn the certain and uncertain data distributions. This loss function is based on kind of penalty term ensuring that the uncertain labels lies in a Wasserstein ball for which the radius is automatically tuned to get the best possible result. This construction crucially relies on the use of the Wasserstein gradient flow associated with Gaussian distributions. To conclude, a series of experiments show that the new methodology proposed in the paper leads to state-of-art results.\n\nI think that the idea is nice and was very impressed by the numerical results reported by the authors. However, it took me a while to just understand the problem and the setting considered by the authors! My second main criticism concerns the theoretical results presented in this paper. While I think that it is important that methods are justified by rigorous results, the one presented in this paper are just not understandable by most people. Two of the main underlying issues in my opinion are that the notion and objects which are used in the paper are not introduced very much rigor (or even not at all) and the result lack of clarity. To be honest, I did not catch half of the sentences of the paper.\n\nI advise the authors to make a in-depth revision of their paper, introducing more carefully their method so it can be understand by a broader audience. One solution in my opinion, is to reduce the theoretical notion and results  to their strict minimum. I think that a lot of them are unnecessary for the introduction of the proposed methodology.   ", "extra_scores": {}, "extra_text": {}}, {"review_id": "0_z3oPosEHg", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Note: initial score was 4; updated to 5 after taking a brief look at the experimental sections; will check again later to take a closer look at the updated \"motivation\" part.\n\n---\n\n__0. disclaimer.__\nIt is possible that I failed to understand a significant portion of this manuscript; I had a very hard time trying to understand the notations and the writing overall. Please correct me if I am wrong.\n\n__1. paper summary.__\nThe paper aims to give a better way of handling the noisy label data, by providing a way to better utilize the information given by _uncertain_ (or high-loss) samples, which are often simply filtered out in previous works. The proposed method is as follows: (1) select the \"lowest-loss mini-batch\" among the _certain samples_ (i.e., low-loss) $Y$, with some empirical measure $\\nu$. (2) Construct an approximate optimal transport map $\\mathcal{F}$ from the measure of uncertain samples $\\mu$ to a Gaussian measure $\\mathcal{N}(\\mathbf{m}_\\nu,\\Sigma_\\nu)$ in the feature space, having the same first and second moments as $\\nu$. (3) Minimize the sum of losses on the certain samples and transported uncertain samples. Authors also introduce several techniques to circumvent\n\n__2. review summary.__\nThe proposed method seems to be advantageous over the considered baselines, but I am not sure if the method would outperform other baselines as well. Also, the clarity of the manuscript is not quite good (in my opinion).\n\n__3. missing baselines.__\nA wide range of algorithms has been proposed recently to make use of the estimated-to-be-mislabeled subset of samples (or uncertain sample, as authors put it) data. For instance, the ICLR 2020 paper by Li et al. ([\"DivideMix: Learning with noisy labels as semi-supervised learning\"](https://openreview.net/pdf?id=HJgExaVtwr)) proposes to drop the labels from the uncertain samples and use semi-supervised learning approaches. The paper also refers to several related approaches, which could/should be compared or discussed against the optimal transport based method proposed by the authors. Other popular baselines would be [INCV](https://arxiv.org/abs/1905.05040) and the beta-mixture modeling by [Arazo et al.](https://arxiv.org/pdf/1904.11238.pdf), which are slightly different in spirit. A recent [NeurIPS 2020 paper](https://arxiv.org/pdf/2001.10528.pdf) (which should be considered as a concurrent work) is worth discussing, for the benefit of the readers.\n\nI strongly recommend adding (at least) an empirical comparison to the Li et al., as it shares a larger goal (in the sense that utilizing the information from mislabeled data) yet implemented with a different philosophy. Understanding when/how one approach outperforms the other should be fantastic.\n\n__4. clarity and (potential) errors.__\nSome parts of the manuscript was not clear enough to me, or perhaps is written in an overly convoluted manner. Clarifying the following bits may help the readers (and me) follow the content more easily.\n- Eq. (2) follows the description \"Specifically, with a probability $1-\\delta$, the generalization bound for clean datasets is defined as follows:\". However, I am not sure whether it is a \"definition of the generalization bound.\" Rather, it should be one of many generalization bounds that one can prove.\n- Eq. (3) contains the expression $\\mathbb{E}_{X\\sim \\mu,\\hat{r}|T} [l(X;\\hat{r})]$. I am not sure if the definition of $T$ appears in the main text. I guess that it is the label transition matrix? Also, on which space is the minimum taken over? Is it over any distribution $\\mu$ on $\\mathbb{R}^d$, or is it more like over any distribution that can be generated by partitioning $\\mathbb{P}$? Any vagueness must be removed, as the authors also provide theoretical results.\n- In definition 1, what distance measure $d(\\cdot,\\cdot)$ are authors using? (also, using same $d$ for distance and dimensions add unnecessary difficulty). If the authors are not using more than two different notions of distances, why don't you just write it directly, like $\\mathbb{E}_{\\mu}\\lVert x_0-x\\rVert_2^2$?\n- In Eq. (5), again it is not clear over what the minimum is being taken over. Is it over all partitioned-distribution $\\mu, \\xi$, or also over all possible transport maps $\\mathcal{F}$?\n\n__5. Rationale__\nI am curious what the general intuition behind the proposed approach is. Why is the \"minimum transportation cost\" map $\\mathcal{F}$ a sensible method to transport the high-loss samples? Are we making any implicit assumptions on the nature of label-flipping operations, or perhaps the learning dynamics itself?\n\n__minor suggestions.__\n- The title can be more informative. I am not sure if the current title \"Wasserstein distributional normalization\" gives any information about the task under consideration, or \n- The notation $\\vartheta$ looks like the model parameter at a first glance. How about changing it to other greek letters?\n- In proposition 1, there should be a typo: $\\mathcal{F}: \\mathbb{R}^+ \\times \\mathcal{P}_2$. The mapping $\\mathcal{F}$ maps to where?", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a potentially very interesting and original approach to handle label noise.  The numerical experiments suggest that the method works very well. But the paper  itself has been deemed very hard and demanding to read and understand for a general machine learning crowd and even by experts in the fields of optimal transport and  Markov theory. 

Note that due to the low confidence in several review an additional emergency review by an expert was asked and it confirmed the global opinion from  other reviewers that the paper is interesting but needs a major rewriting before acceptance in a ML conference. The AC strongly suggest that the authors work on a more pedagogical introduction and explanation of the method before resubmitting.  
Decision: Reject