Provable Robust Learning for Deep Neural Networks under Agnostic Corrupted Supervision
Authors: Boyang Liu
Mengying Sun
Ding Wang
Pang-Ning Tan
Jiayu Zhou
Venue:  (2021)
Keywords: Noisy Label
Corrupted Supervision
Robustness
Optimization

Abstract: Training deep neural models in the presence of corrupted supervisions is challenging as the corrupted data points may significantly impact the generalization performance.  To alleviate this problem, we present an efficient robust algorithm that achieves strong guarantees without any assumption on the type of corruption and provides a unified framework for both classification and regression problems. Different from many existing approaches that quantify the quality of individual data points (e.g., loss values) and filter out data points accordingly, the proposed algorithm focuses on controlling the collective impact of data points on the averaged gradient.  Even when a corrupted data point failed to be excluded by the proposed algorithm, the data point will have very limited impacts on the overall loss, as compared with state-of-the-art filtering data points based on loss values. Extensive empirical results on multiple benchmark datasets have demonstrated the robustness of the proposed method under different types of corruption.

Under review as a conference paper at ICLR 2021
PROVABLE ROBUST LEARNING FOR DEEP NEURAL NETWORKS UNDER AGNOSTIC CORRUPTED SUPERVI- SION
Anonymous authors Paper under double-blind review
ABSTRACT
Training deep neural models in the presence of corrupted supervisions is chal- lenging as the corrupted data points may significantly impact the generalization performance. To alleviate this problem, we present an efficient robust algorithm that achieves strong guarantees without any assumption on the type of corruption. and provides a unified framework for both classification and regression problems. Different from many existing approaches that quantify the quality of individual data points (e.g., loss values) and filter out data points accordingly, the proposed algorithm focuses on controlling the collective impact of data points on the av- eraged gradient. Even when a corrupted data point failed to be excluded by the proposed algorithm, the data point will have very limited impact on the overall loss, compared with statc-of-the-art filtering data points based on loss values. Extensive empirical results on multiple benchmark datasets have demonstrated the robustness of the proposed method under different types of corruptions.
INTRODUCTION
Corrupted supervision is common issue in real-world learning tasks, where the learning targets are not accurate due to various factors in the data collection process. In deep learning models, such corruptions are especially severe, whose degree-of-freeddm makes them easily memorize corrected examples and susceptible to overfitting (Zhang et al. 2016).
There are extensive efforts achieve robustness against corrupted supervisions. A natural approach to deal with corrupted supervision in deep neural networks (DNNs) is to reduce the model exposure corrupted data points during training. By detecting and filtering (or re-weighting) the possible corrupted samples, the learning is expected to deliver a model that is similar to the one trained on clean data (without corruption) (Kumar et al., 2010; Han et al.. 2018 Zheng et al.) 2020). There are different criteria designed to identify the corrupted data points in training. For example, Kumar et al. loss function values data points; Zheng et al. (2020) tapped prediction uncertainty for filtering data; Malach & Shalev-Shwart (2017) ) used the disagreement between two deep networks: Reed et al. (2014) utilized the prediction consistency of neighboring iterations. The success of these methods highly depends on the effectiveness of the detection criteria in correctly identifying the corrupted data points. Since the corrupted labels remain unknown throughout the learning, such "unsupervised" detection approaches may not be effective, either lack theoretical guarantees of robustness (Han et al., [0018] Reed et al.) 2014 Malach &
(2010) Han et al. 20188; Jiang al. (2018) leveraged
Shalev-Shwartz 2017 Li et al.) 2017) provide guarantees under assumptions of the availability of prior knowledge about the type of corruption (Zheng et al. 22000) Shah et al. 2020 Patrini et al., 2017 Yi & Wu 2019). Besides, another limitation of many existing approaches is that. they are exclusively designed for classification problems (e.g., Malach & Shalev-Shwartz (2017); Reed et al. 2014); Menon al. (2019): Zheng et al. (2020)) and are not straightforward to extend to solve regression problems.
To tackle these challenges, this paper presents a unified optimization framework with robustness guarantees without any assumptions how supervisions are corrupted, and is applicable to both classification and regression problems. Instead developing an accurate criterion for detection cor- rupted samples, we adopt a novel perspective and focus on limiting the collective impact of corrupted samples during the learning process through robust mean estimation of gradients. Specifically, our estimated average gradient is close to the gradient from the clean data during the learning iterations,

---

Under review as a conference paper at ICLR 2021
then the final model will be close to the model trained clean data. As such, a corrupted data point can still be used during the training when it does not considerably alter the averaged gradient. This observation has remarkably impact our algorithm design: instead of explicitly quantifying (and identifying) individual corrupted data points, which is a hard problem in itself, we are now dealing with easier task, i.e., eliminating training data points that significantly distort the mean gradient estimation. One immediate consequence of this design is that, even when a corrupted data point failed to be excluded by the proposed algorithm, the data point is likely have very limited impact on the overall loss, as compared with state-of-the perform experiments on both regression and classification with corrupted supervision on multiple benchmark datasets. The results show that the proposed method outperforms state-of-the-art.
filtering data points based on loss values. We
2 BACKGROUND
Learning from corrupted data (Huber 1992) has attracted considerable attention in the machine learning community (Nataaajan et al. 2013). Many recent studies have investigated robustness of classification tasks with noisy labels. For example, Kumar et al.] (2010) proposed self-paced learn- ing (SPL) approach, which assigns higher weights to examples with smaller loss. A similar idea was used in curriculum learning (Bengio et al., 2009), in which the model Iearns casy samples first before learning harder ones. Alternative methods inspired by SPL include learning the data weights (Jiang et al. 2018) and collaborative learning (Han et al.) 2018 Yu et al. 2019). Label correction (Patrini al.] [217]] Li et al., [017 Yi & Wu [019) is another approach, which revises original labels in data with goal to recover clean labels from corrupt ones. However, since we do not have
access to which data points are corrupted, it is hard get provable guarantees for label correction
without strong assumptions regarding the corruption type.
Accurate estimation of gradients is a key step for successful optimization. The relationship between gradient estimation and its final convergence has been widely studied in the optimization commu- nity. Since computing an approximated (and potentially biased) gradient is often more efficient than computing the exact gradient, many studies used approximated gradients to optimize their models and showed that they suffer from the biased estimation problem there is no assumptions on the gradient estimation (d' Aspremont 2008 Schmidt et al. 2011 Bernstein et al. 2018 Hu et al. 2020 Ajalloeian & Stich 2020).
A closely related topic is robust estimation of the mean. Given corrupted data, robust mean estima- tion aims at generating an estimated mean á such that the difference between the estimated mean on corrupted data and the mean of clean data || /lll2 is minimized. It was showed that median or trimmed-meat are the optimal statistics for mean estimation in -ineeinnsio data (Huber) 1992). However, robustness in high dimension is quite challenging since applying the coordinate- wise op- error factor ((vd) that scales with the data dimension.
timal robust estimator would lead Although some classical work, such as Tukey median (Tukey 1975), successfully designed algo- rithms to get rid of the (vd) error, the algorithms themselves are not polynomial-timg algorithm. More recently, Diakonikolas al. (2016); Lai et al. (2016) successfully designed polynomial-time algorithms with dimension-free error bounds. The results have been widely applied to improve algorithmic efficiency in various scenarios (Dong et al. 2219:; Cheng et al.) 2020).
Robust optimization aims to optimize the model given corrupted data. Many previous studies im- prove the robustness of the optimization in different problem settings. However, most of them either study linear regression and its variantes/ (Bhatia et al. 2015 2017: [Shen & Sanghavi 2019) or study the convex optimization (Prasad et al.) 2018). Thus, those results cannot be directly generalized to deep neural networks. Diakonikolas et al. (019 is very generalized non-convex optimization method with the agnostic corruption guarantee. However, the space complexity of the algorithm is high, thus cannot be applied to deep neural networks given current hardware limitations.
3 METHODOLOGY
Before introducing our algorithm, we first discuss the corrupted supervision. To characterize agnos- tic corruptions, make use of adversary that tries to corrupt the supervision of clean data. There is no limitation on how the adversary corrupts the supervision, which can either be randomly permuting the target, or in way that maximizes the negative impact (i.e., lower performance).
2

---

Under review as a conference paper at ICLR 2021
Firstly, the adversary choose up to fraction of the clean target Dj Rax; and change the selected row of Dy to arbitrary valid numbers, generating Dú E I=*9, Then, the adversary returns the corrupted dataset D., D to our learning algorithm A. In this process. the only constraint on the adversary is the fraction, and the adversary has full knowledge of the data, and even the learning algorithm A. A natural question
ask is: Given data set with e-fraction corrupted supervision D2 E _x D and a learning objective 6 IRP X IR x IR + IR parameterized by 0, can we output parameters O R- such that [V Dz. Dy)ll is minimized.
When e = O, we have D; = Dy and learning is done on the clean data. The stochastic gradient descent could converge to a stationary point, where (oV Da, D2)]l O. However, when the supervision is corrupted as above, this is not the case any more, due to the error in 0 impacted by the corrupted data. We thus want an efficient algorithm to find a model that minimizes (VV D2, Dy)ll- A robust model 0 should have small value of VVo Dz, Dy) || and we hypothesize that a smaller (VV Dz, Dy)ll has better generalization.
3.1 STOCHASTIC GRADIENT DESCENT WITH BIASED GRADIENT
A direct consequence of corrupted supervision is biased gradient estimation. In this section, we will first analyze how such biased gradient estimation affeets the robustness of learning. The classical analysis of stochastic gradient descent (SGD) requires access to the stochastic gradient oracle, which is an unbiased estimation of the true gradient. However, corrupted supervision leads to corrupted gradients, and it is thus difficult to get unbiased gradient estimation without assumptions of how the gradients are corrupted. We start the analysis by the following informal theorem (without elaborated discussions of assumptions) of how biased gradient affects the final convergence of SGD, Its formal version is provided in Theorem Appendix.
Theorem (Convergence of Biased SGD (Informal)) Under mild assumptions, denote Ç to be the maximum E norm of the difference between clean minibatch gradient and corrupted minibatch gra- dient g - Ell S Ç. then by using biased gradient estimation, SGD converges to the <-approximated stationary points: E| (1.)) = )(
Remark In the corrupted supervision setting, let the gradient estimated by corrupted data De be g. the gradient estimated by clean data D be g. Assume | - gl S S. it follows that when using corrupted dataset in SGD, converges to the {-approximated stationary point of the objective defined by the clean data. Note the difference between above theorem and rypical convergence theorem is that we are using biased gradient estimation.
According to Theorem I and the remark, robust estimation of the gradient is the key to ensure a robust model (converge the clean solution). We also assume the loss function has the form of f(y, $), where many commonly used loss functions fall in this category.
3.2 ROBUST GRADIENT ESTIMATION FOR GENERAL DATA CORRUPTION
We first introduce Algo. 2 for general corruption (i.e. corruption on both features and/or supervi- sions). The algorithm excludes the data points with large gradient norms, and uses the empirical mean of the remaining to update gradients. In Thm. 2 we give its robustness property.
Algorithm/ 1: Robust Mean Estimation for Corrupted Gradient
gradient matrix G E m x d, corruption rate return estimated mean ja Rd
input:
For each row z; in G. calculate the 12 norm ||a,]||
2. Choose the e- e-faction rows with large ||z]]|
3. Remove those selected rows, and return the empirical mean of the rest points as fo.
Assumption (Individual L-smooth loss) For every individual loss function di there exists con- stant L > O, such that for a clean sample i, we have (Øx(x) - ox(y)) < L L'x y) for any y.
Theorem (Robust Gradient Estimation For Data Corruption) Let G E Rmmd be a corrupted gradient matrix, and G x be the clean gradient mairix. Let μ be the empirical mean function,
3

---

Under review as a conference paper at ICLR 2021
Algorithm 2: (PRL(G)) Provable Robust Learning for General Corrupted Data
input:
Label corrupted dataset D2, Dy, learning rate 7/6
return model parameter 0;
for == I to maxiter do
Randomly sample a minibatch M from Dz. D
Calculate the individual gradient G for M Apply Algorithm 1 on G get robust gradient estimation Update model 0.+1 = 0, - "/1p
end
we have that the output of Algo. ف of G satisfies [|p(G) - Alll = O(evd). Moreover, if Asm. holds, we further have || ()) - μ O(LL).
Combining with the aforementioned convergence analysis biased SGD, we get the following:
Corollary (Robust Optimization For Corrupted Data) Given assumptions used in Thm. 7 and
Asm. D applying Algo. [to any e-fraction corrupted data, we get minkelT] E| Vo(x,)] (eL))
with large enough T. If Asm. does not hold, then we get minterry [[|Vo(())] O(ev d) with large enough T.
The robustness guarantee states that even training on generally corrupted data (corrupted supervision is special case), Algo. 2 guarantee that the gradient norm on remaining data cannot be too large. Since Thm. 2 gives a dimension-free error bound when Asm. [ holds, Corollary [ also gives the dimension-frees robustness guarantee with Asm. We defer the detailed discussion of (e to later sections. Although the error bound () sounds good, we note that it still has several drawbacks:
First, the dimension-free error bound means the error does not grow with increasing dimensions, and critical when working with neural networks, due to the extremely large gradient dimension (i.e., #parameters of neural network). Thm. 2 gives the dimension- free error bound only when Asm. holds, which is quite strong. In addition, even when Asm. hoolds, can be large, leading to large gradient estimation error. Existing work (Diakonikolas al. 2019) already acheives the dimension- free (vvc) guarantee with general corruptions, which is a much more better theoretical results than above theorem. However, in practice, we found that the gradient norms of deep neural networks for individual data points are usually not very large, even at the beginning of the training. This can be partially due to the network structure. Further discussion on this issue is beyond the scope of this paper. but the theoretical bound above states that the robustness should depend on the number of parameters for the general models.
Another concern of Alg. is the efficiency. It requires computing individual gradients. Although there are some advanced approaches to get the individual gradient, e.g., (Goodfellow) 2015). it is still relatively slow as compared to commonly used paek-propggation Moreover, these methods are usu- ally not compatible with popular components such as batch normalization (BN) since the individual gradients are not independent inside BN, using of which will lose the benefits from parallelization.
3.3 ROBUST GRADIENT ESTIMATION FOR ONE DIMENSIONAL CORRUPTED SUPERVISION
In this section, we show that the above robustness bound can be improved if we assume the corrup- tion only comes from supervision. Also, by fully exploiting the gradient structure of the corrupted supervision, our algorithm can be much more efficient and meanwhile compatible with batch nor- malization. We use the one dimensional supervision setting (binary classification or single-target regression) to illustrate this intuition and extend it more general settings in the next section. Con- sider a high- sigh--imensional supervised learning problem with X E RT*P and y E R". The goal is to learn function f parameterized by 0 E d minimizing the following loss minş En di Øl, 8f. a,g,
ming Ei Clys, fxx 8)) The gradient for data point is VoΦ, af. 86
One key observation is that: when only supervision corrupted, then the corruption contributes 8r, which is a scalar in the one-dimensional setting. In other words, given only the term a; at. the clean gradient of 8th point, 9: E Rd, the corrupted supervision can only perturbs the the length of the gradient vector, changing the gradient from ogg; to dggi: where 8; 35.
ai; When og and 6, are
4

---

&4. which is Under review as a conference paper at ICLR 2021
known, then we can easily eliminate the impact from comupttd supervision. But this is not the case since we have have only the possibly corrupted target OJ as opposed to the ground truth yi-
On the other hand, the fact that corrupted supervision scales the clean gradient can be used to reshape the robust optimization problem. Recall that in every iteration, we update our model by 0 = 0 - 7-4 G), where μ denotes the empirical mean function and G = [Vool ..... Voo Rmxd is the gradient matrix with mini-batch size m. We then have the following:
Problem 1 (Robust Gradient Estimation for Corrupted Supervision - One Dimensional Case)
Given clean gradient matrix G E Rmxxd e-corrupted matrix G with at most e-fraction rows are corrupted from ogg; to igga design an algorithm A: II + R that minimizes ||{(G))A(())
Note that when || is large, the corrupted gradient will have large effect on the empirical mean, and vice versa. This motivates us to develop an algorithm that filters out data points by the loss layer gradient || @L_ ||. If the norm of the loss layer gradient of a data point is large (in one-dimensional case, this gradient reduces to a scalar and the norm becomes its absolute value), we exclude the data point when computing the empirical mean of gradients for this iteration. Note that this algorithm is applicable to both regression and classification problems. Especially, when using the mean squared error (MSE) loss for regression, its gradient norm is exactly the loss itself, and the algorithm reduces to self-paced learning Kumar et al. (2010). We summarize the procedure in Alg. 3 and extend it to the more general multi-dimension case in the next section.
Algorithm 3: (PRL(L)) Efficient Provable Robust Learning for Corrupted Supervision
input: dataset D,, D with comupted supervision, learning rate N/: return model parameter 0; for / to maxiter do
Randomly sample minibatch M from D.,, D2
Compute the predicted label Y from M
Calculate the gradient norm for the loss layer, (i.e. 118 y/ for mean square error entropy) for each data point M
Remove the top r-fraction data from M according to |l - -yl Return the empirical mean of the remaining M as the robust mean estimation f
Update model 0.+1 = 0. "tef
end
3.4 EXTENSION TO MULTI- MULII-DIMENSIONAL CORRUPTED SUPERVISION
To extend our algorithm and analysis to multi- mulli-dimensional case, let to be the supervision dimen- sion, the gradient for each data point is Ved; = Øl: 8f, where 0i; E IR is the gradient of loss 0f:
af: 80
respect to model outputs, and 0f: E Rexd is the gradient of model outputs respect to model parame- 06 ters. Similarly, when the supervision is corrupted, the corruption comes from the term öf.
aL RV, W; af. Raxd m be the minibatch size. Denote vector. Let 6; उ.. E R9, ai क7.
85
30
the clean gradient matrix G E Rm where the ith, row of gradient matrix g, = a, W, Now the robust gradient estimation problem is defined by:
Problem 2 (Robust Gradient Estimation for Corrupted Supervision - Multi--iienniiona Case) Given a clean gradient matrix G, an e-corrupted matrix G with at most e-fraction rows are cor- rupted from a; W; to 6, W., design an algorithm A [g + Rdd that minimizes [[[[G) A(G)]
We start our analysis by investigating the effects of the filtering base algorithm, i.e. use the empirical mean gradient of (1 ff-fraction subset to estimate the empirical mean gradient of clean gradient matrix. We have the following for a randomized filtering-base filterri based ddgorithm(proot in Appendix):
Lemma 1 (Gradient Estimation Error for Random Dropping c-fraction Data) Let G Bxxd be a corrupted matrix generated as in Problem 2 and G E R xd be the original clean gradient matrix. Suppose arbitrary (1 c) e)-fraction rows are selected from G to form the matrix N E Ruxd Let p be the empirical mean function. Assume the clean gradient before loss layer has bounded
5

---

Under review as a conference paper at ICLR 2021
operator norm, i.e., | C, the maximum clean gradient in loss layer max4e ||aو|| k, the maximum corrupted gradient in loss layer maxçEN ||+1| = U, then we have:
[l((G) MNN) S Ck 3E - 4c2
Cu 1-6
1-6
We see that is the only term that is related to the corrupted supervision. If is large, then the bound is not safe since the right-hand side can be arbitrarily large (i.e. an adversary can change the label in a way such that v is extremely large). Thus controlling the magnitude of e provides way to effectively reduce the bound. For example, if we manage control v S k. then the bound is safe. This can be achieved by sorting the gradient norms at the loss layer, and then discarding the largest e-fraction data points. We thus have the following result.
Theorem (Robust Gradient Estimation For Supervision Corruption) Let G be corrupted matrix generated in Problem be the label dimension, μ be the empirical mean of clean matrix G. Assume the maximum clean gradient before loss layer has bounded operator norm: [Wllop S C,
then the outpul of gradient estimation in Algo 3 is satisfies || jol ( R (
Compare Thm. 2 and Thm. we see that when the corruption only comes from supervision, the dependence on d is reduced q. where in most deep learning cases we have d n. Applying Thm directly shows that our algorithm is also robust in multi-label settings.
3.5 COMPARISON WITH DIAKONIKOLAS ET AL.. (019 AND OTHER METHODS
SEVER Diakonikolas et al. 2019) showed promising state-of-the-ar theoretical results in general corruptions, which achieves O(vc)
dimension- -oeeee guarantee for general corruptions. Compared to Diakonikolas et al. (2019), we have two contributions: a). By assuming the corruption comes from the label (we admit that this is quite strong compared to the general corruption setting), we could get a better error rate. b). Our algorithm can be scaled to deep neural networks while Diakonikolas ct al. (2119) cannot. We think this is a contribution considering the DNN based models are currently oatt-oofthheaat methods for noisy label learning problems (at least in empirical performance).
Although Diakonikolas et al. 2019) achieves very nice theoretical results, unfortunately, it can- not be applied to DNN with the current best hardware configuration. Diakonikolas al. (019) uses dimension-free robust mean estimation breakthroughs to design the learning algorithm, while
notice that most robust mean estimation relies on filtering out data by computing the score of projection to the maximum singular vector. For example, in Diakonikolas et al. (2019), it requires performing SVD on d individual gradient matrix, where the sample size and is the number of parameters. This method works well for small datasets and small models since both n and is small enough for current memory limitation. However, for deep neural networks, this matrix size is far beyond current GPU memory capability. That could be the potential reason why in Diakonikolas al. (2019), only ridge regression and SVM results for small data are shown (we are not saying that they should provide DNN results). In our experiment, our is 60000 and d is in the magnitude of millions (network parameters). is impractical to store 60000 copies of neural networks in a single GPU card. In contrast, in our algorithm, we do not need to store the full gradient matrix. By only considering the loss-layer gradient norm, we can easily extend our algorithm to DNN, and we showed that this simple strategy works well in both theory and challenging empirical tasks.
We notice that there are some linear (Bhatia et al. 2015 2017) or convex method (Prasad et al. 2018) achieves the better robustness guarantee. However, most of them cannot be directly applied to deep neural networks.
4 RELATIONSHIP TO SELF-PACED LEARNING (SPL)
SPL looks very similar to method at first glance. Instead of keeping data point with small gradient norm, SPL tries to keep data with small loss. The gradient norm and loss function can be tied by the famous Pojsak-Eojasiewicz (PL) condition. The PL condition assumes there exists some constant s > such that V[( 2 s (ø(x) o²). Yx holds. As we can see, when the neural network is highly over-parameterizze, the d² can be assumed to be equal across different

---

Under review as a conference paper at ICLR 2021
samples since neural networks can achieve 0 training loss (Zhagg et al. 2016). By sorting the error o(x,) for every data point, SPL actually is sorting the lower bound of the gradient norm the PL condition holds. However, the ranking of gradient norm and the ranking loss can be very different since there is no guarantee that the gradient norm is monotonically increasing with the loss value. We provide illustration of why SPI. is not robust from geometric perspective in the appendix. Here we show even for simple square loss, the monotonic relationship is easy to break. One easy counter-example
((11,,z)) = 0.5xî + 50z Take two points (1000, 1) and (495, 49.5), we will find the monotonic relationship does not hold for these two points. Nocedal et al. 22022) showed that the monotonic relationship holds for square loss (i.e.((x) = 2 (x-xx Q(x
x") ) if the condition number of Q is smaller than 3 + 2v2, which is a quite strong assumption especially when x is in high-dimension. If we consider the more general type of loss function (i.e. neural network), the assumptions on condition number should only be stronger, thus breaking the monotonic relationship. Thus, although SPL sorts the lower bound of the gradient norm under mild assumptions, our algorithm is significantly different from the proposed SPL and its variations.
Now, we discuss the relationship between SPL and algorithm under supervision corruptions. SPL has the same form as algorithm 3 when we are using mean square error to perform regression tasks since the loss layer gradient norm equal to loss itself. However, in classification, algorithm is different from the SPL. In order to better understand the algorithm, we further analyze the difference between SPL and our algorithm for cross-entropy loss.
For cross entropy, denote the output logit as o, we have Hy: f.) -,, (xxsssss)(axxx()))))
-y,,, log(E))) The gradient norm of cross entropy w.r.t o, is: ÖH, = y. - softmax(o.) = f, -yy do:
Thus, the gradient of loss layer is the MSE between and f. Next, we investigate when MSE and Cross Entropy gives non-monotonic relationship. For the sake of simplification. we only study the sufficient condition of the non-monotonic relationship, which is showed in lemma 2
Lemma 2 Let y E IRS, where Yı- = 1, 0 for i + k, and ax 3 are two q dimensional vector in probability simplex. Without loss of generality, suppose a has smaller cross entropy loss ork Z 3k. then the sufficient condition for ||a yll > || - is Var ((a,)) - (Vrr+x((33)) 2
9 (lax - -2 - as - -Bk))
1-11
As 0% 2 Br. the right term non non-negative In conclusion, when MSE generates a differ- ent result from cross-entropy, the variance of the probability of the non-true class of the dis- carded data point is larger. Suppose we have ground-truth vector y = 00, 0, O, 0, 0, O, 0, O, Oj. [0.08, 0.28, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08, 0.08] and = [0.1, 0.3, 0.34, 0.05, 0.05, 0.1, 0.03, 0.03, 0, 0j. The prediction a have a smaller mse loss while prediction / have a smaller cross-entropy loss. It is intuitive that /8 is more likely to be noisy data since it has two peak on the prediction (i.e. 0.3, 0.34). However, since cross entropy loss only con- siders one dimension, it cannot detect such situation. Compared to the cross-entropy, the gradient (mse loss) considers all dimension, and thus will consider the overall prediction distributions.
and we have two predictions
5 COMBINING WITH CO-TEACHING STYLE TRAINING
Motivated by co-teaching (Han et al., 2018), which is one of currently state-of-the-ar deep meth- ods for learning under noisy label, we propose Co-PRL(L), which has the same framework of co- teaching but uses the loss-layer gradient to select the data. The full algorithm is shown in algorithm in the appendix. The meaning of all original (Han et al. (2018). Compared with algorithm 3 except sampling data according to the loss layer gradient norm, the Co-PRL(L) has two other modules. The first is we gradually increase the amount of the data to be dropped. The second is that two networks will exchange the selected data to update their own parameters.
in algorithm 4 are all the same as in the
6 EXPERIMENT
In this section, we perform experiments on benchmark regression and classification dataset. The code is available in supplementary materials of submission. We compare (Algo) 2 PRI(L.)
7

---

Under review as a conference paper at ICLR 2021
(Algo. and Co-PRL(L) (Algo. to the following baselines. Standard: standard training with- out filtering data (mse for regression, cross entropy for classification); Normelip: standard training with norm clipping; Huber: standard training with huber loss (regression only); Decouple: decou-
pling network, update two networks by using their disagreement (Malach & Shalev-Shwartz/ (017)
(classification only); Bootstrap: It uses weighted combination of predicted and original labels as the correct labels, and then perform back propagation (Reed et al., 2014) (classification only);
Min-sgd: choosing the smallest loss sample in minibatch to update model (Shah et al. 2020); SPL:
self-paced learning, dropping the data with large losses (same as PRL(L) in regression setting with MSE loss): Ignormclip: clipping individual gradient then average them to update model (regression only); Co-teaching: collaboratively train pair of SPL model and exchange selected data to another Han et al. (218)) (classification only); is hard to design experiments for agnostic corrupted supervision and we tried our best to include different types of supervision noise. The supervision corruption settings are as follows: linadv: the corrupted supervision is generated by random wrong linear relationship of features (regression); signflip: the supervision sign is flipped (regression); uninoise: random sampling from uniform distribution as corrupted supervision (regression); mix- ture: mixture of above types of corruptions (regression); pairflip: shuffle the coordinates (i.e. eyes to mouth in celebA or cat to dog in CIFAR) (regression and classification); symmetric: randomly assign wrong class label (classification). For classification, we use classification accuracy as the evaluation metric, and R R-square is used to evaluate regression experiments. Due to the limit of the space, we only show the average evaluation score on testing data for the last 10 epochs. The whole training curves are attached in the appendix. All experiments are repeated 5 times for regression ex- periments and 3 times for classification experiments. Main hyperparameters are showed in appendix.
6.1 REGRESSION EXPERIMENT
We use CelebA data to perform regression tasks. CelebA dataset has 162770 training images, 19867 validation images, 19962 test images. The target variable is ten-dimensional coordinates of the left eye, right eye, nose, left mouth, and right mouth. Given the human face image, the goal is to predict 10 face landmark coordinates in the image. We tried adding different types of noise on the landmark coordinates. We preprocess the CelebA data as following: we use three-layer CNN to train 162770 training images to predict clean coordinates (we use 19867 validation images to do the early stopping). Then, we use well-trained network to extract the 512-dimensional feature on testing sets. Thus, our final data to perform experiment has feature sets X IR 19962x512 and the target variable Y E RR 119962210 We further split the data to the training and testing set, where training sets contain 80% of the data. Then, we manually add linadv, signflip, uninoise, pairslip, mixture types of supervision noise on the target variable on training data. The corruption rate for all types of corruption is varied from 0.1 0.4. We use 3-layer fully connect networks in experiments. The results of averaged last 10 epoch r-square are in table []
6.2 CLASSIFICATION EXPERIMENT
We perform experiments CIFARIO. 10, and CIFARIO0 to illustrate the effectiveness of our algorithm in classification setting. We use the 9-layer Convolutional Neural Network, which is the same as Han et al. 2018). Since most baselines include batch normalization, it is difficult to get individual gradient efficiently, we will drop the ignormelip and PRL baselines. In the appendix, we attached the results if both co-teaching and Co-PRL(L) drops batch normalization module. We will see that co- teaching cannot maintain robustness while our method still has robustness. The reason is discussed in the appendix. We consider pairflip and symmetric supervision corruptions in experiments. Also, to compare with the current state of the art method, for symmetric noise, we use corruption rate which beyond 0.5. Although our theoretical analysis assumes the noise rate is small than 0.5, when the noise type is not adversary (i.e. symmetric), we empirically show that our method can also deal with such type of noise. Results on CIFARIO. CIFARIOR are in Table 2 As we can see, no matter using one network (PRL vs SPL) two networks (Co-PRL(L) vs Co-teaching), our method performs significantly better. Since in real-world problems, it is hard to know that the ground-trutt corruption rate, we also perform the sensitivity analysis in classification tasks to show the effect of overestimating and underestimating e. The results are in Table 3 More discussion about sensitivity analysis can be found in appendix.
8

---

Under review as a conference paper at ICLR 2021
Corsuption
Standard
Normelip
Huier
Min
lenoemelip
PRLJO)
FRLI
Co PRUZI
linady:
0.
2.22200.7
0.6
0.100-40003
0.68
0.076:0001
0.8644001
0.876=0.01
linadr: 20
3..55#2.1
0.17=00015
0.120
0.367
1871=00.0
0.809+000
0.365=0001
linall: 50
-18.529 2..0044.444 (44
-19188::4.11
0,592=0.77
0. 0.144-0003
4.84420511
0.86==0001
0.861
0.60
72.7517077
2..22 22
000001000
0.00
0.857
0000
0.07
sigullip: 10
0.000:00000
0.708:0003
0.87=0001
0.110:0004
0.84620001
087720001
0.878-00001
0.89
sgnillip: 20
0.611 1008
0.301000
0.006
0.10 1000
08220.02
0.875
0.075
0.77
sigullip:
04
0,41 20004
0.629:0003
0.
0.705:0:02
0:81
0.871::000
0.805
signllip: 40
0.193-10000 1:00333
0.19010004
0.309
00028 0:25
0.75
0,872 0.01
0.821 0.01
0001
unincrse:
1.000
0000
0,05
1:000
0.00
0.00
0,881
unincuse: 20
0.798
F000
0.79
F.02
0.04
0.878:0.01
0 0.01
0.380 0.01
uninctse: 30
0.02
0,725
0.00
0.14640000
0:00
0.876:0001
0.69900011
0.79100001
uninosse: 40
0,054 4002
0:25=0000
0.000
0.821
0.878
0,884
partlip: 10
1.02
1002
0.00
0.1001000
0.86/20:01
0.8710.01
0.876100001
0.001
parffip: 20
0778440000
0.785-0005
0,841
+0,02
0.120400000
0.899-000
0,874 0.01
0.0734-0000
0.574 0.001
partlips 30
1 F000
0.770 0.02
0.00
0.826 0.00
0.00
0.8211000
0
partfiy 40
0.5640006
0,555-0006
0.00
0.13440000
0:10==0:02
0.865=0000
0.87010.01
0.8700
misture:
02024000
0.73=0001
10000
0:00
0.875
0,850
0,880
mixture: 20
0..04
0.66
0,855
0.855=0:01
0.11940000
0.855=0000
0.877
0.874+0000
0.879=0.01
0.716
0.00
0.148
F000
0.:7
0,875
0.01
0.8774000
0 0.01
mixtire:
1.100:1511
-2000:0044
0.763=0001
0.07544002
0.85=0001
0.72=0.01
0,875 #0.01
0.866:0001
Table 1: CelebA Results. The numbers are r-square on clean testing data, and the standard deviation from last ten epochs and random seeds.
Coruption
Standadd 63.22 + 0.18
Normelip
Booksrap
Decouple
spd
PRL
Losscaching
Co
CPI0-sym-30
62:41 + 0.06
63.67
0.24
70.73 4 0.51
13:31 + 2:24
77.77 + 0.34
79:40 = 0.19
79.00 + 0.13
50,05 + 0.12
CP10 sym 50
48.83
1 0.18
43.99 1 0.28
46.13
0.18
57,48 1: 1,98
13:33 1 2.85
72.22 0.0.5
74.17 3: 0.15
74.25 1 0.41
75.43
0.09
CFI0-sym-70
24.12
01.09
24.17 + 0.37
25.13
40.11
4.62
0.94
56.19
58.36
0.62
58.11
60.26
pt-2-pt-25
68.34 4: 0.30
67.92 + 0.43
08.71
0.32
75,59 + 0.35
10.45
4 0.60
73.79 +: 0.44
50,54 + 0.07
80.18 4 021
$1.51 +: 0.13
CF10-pf-35
0.18
58.19
0.44
12.29
70.40
77.97
CP10-p1-45
48.05 + 0.25
48003 + 0.34
47.84
0.32
51.34 + 031
1094
+ 1.28
58.95 + 0.59
71.42 + 0.24
72.43 4 0.11
73.78
0.17
(F100-8y--9) 30
32.83 12.83:0000 0.39
32.10 4: 0.64
14.47
= 0.22
32.95 + 0.44
2.04 194:000 0.1
44.37 2: 0.44
46.40 = 0.18
45.02 4 0.20
47.51 = 0.47
CF100-qya-50
20.27
0.44
19.33 +: 0.29
21.59
21,02
044
37.89 17.900000
0.65
58.79 0.33
40.64
CF100-8ym70 70
9.93 = 0.07
9:03 + 0.33
10:59
0.17
12.55 0.46
2.32 0 034
24.10 0:44
28 0.56
24.94 0:51
27.27 + 0.01
CF100-p6-25
40.57
+ 0.35
40.22
39,43 4 0.27
0.36
20.48
47.57
2 0.37
42.97 0:10
48,06 0.36
CF100-96-33
34007 0.19
32.88
+ 0.10
34.53= 0.23
33.14 0.07
2.30
2: 0.07
34.17:00.46
43.37 0.16
36.69
0.3
44.08
0.33
CF100-pP-4 4/-45
27.56 1 0.50
27.35
1 0.61
27.56
0.25
26.83
1: 0.41
235
1 032
27.55 1 0.06
33.31 0.10
29.71 1 0.00
34.43
1 0.05
Table 2: Classification Results CIFARIO and CIFARI00 for symmetric and pairflip label cor- ruption. The numbers are classification accuracy on clean testing data, and the standard deviation is from last ten epochs and random seeds.
Daaa
0.1
0.03
c 0.05
0.1
CF10-Piir45%
63
700720667
77
35.56:0055
79.36=0:43
CF10 Sym 50%
09.21
125340445
75:43 1: 008
77.65
78,10=0.31
Sy 70%
0.66
58491097
60.26
60
0..6
CF100 Pair 45%
02.00-00.45
34.1740.40
35,43 0008
36.77041
0.73
CF100 Sym-50%
07.74:041
39.7210.35
00.61 01
40.0210366
1392-0.8
Cy100-Sym-- 70%
2:40
23.50
27.27 0.10
20.00-0300
Table 3: sensitivity analysis for estimated
CONCLUSION
In this paper, we proposed efficient algorithm to defense against agnostic supervision corruptions. Both theoratical and empirical analysis showed the effectiveness of our algorithm. There are two remaining questions in this paper which deserves study in future. The first one is whether we can further improve Occ) error bound or show that O(c) is tight. The second one is to utilize more properties of neural networks, such as the sparse gradient, to see whether it possible to get better algorithms.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Ahmad Ajalloeian and Sebastian U Stich. Analysis of sgd with biased gradient estimators, arXiv preprint arXiv:2008. 1000:2008000001,,, 2020.
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th anmual international conference on machine learning. pp- 41-48, 2009,
Jeremy Bernstein, Yu-X Yu-Xiang Wang, Kamyar Azizzadenesheli, and Anima Anandkumar. signsgd: Compressed optimisation for non-convex problems. arXiv preprint arXiv:1802.044333 044444 2018
Kush Bhatia, Prateek Jain, and Purushottam Kar. Robust regression via hard thresholding. In Advances in Neural Information Processing Systems, pp. 721-729, 2015.
Kush Bhatia, Prateek Jain, Parameswaran Kamalaruban, and Purushottam Kar. Consistent robust regression. In Advances in Neural Information Processing Systems, pp. 2110-2119, 2017.
Yu Cheng, Ilias Diakonikolas, Rong Ge, and Mahdi Soltanolkotabi. High-dimensional robust mean estimation via gradient descent. arXiv preprint arXiv:2005.01378, 2020.
Alexandre d' Aspremont. Smooth optimization with approximate gradient. SIAM Journal on Opti- mization, 191-)117 2008.
Olivier Devolder, François Glineur, and Yurii Nesterov. First-order methods of smooth convex optimization with inexact oracle. Mathematical Programming, 146(1-2)377-5, 2014.
Ilias Diakonikolas, Gautam Kamath, Daniel M Kane, Jerry Li, Ankur Moitra, and Alistair Stewart. Robust estimators in high dimensions without the computational intractability. In 2016 IEEE 57th Annual Symposium on Foundations of Computer Science (FOCS), pp. 655-664 IEEE, 2016
Ilias Diakonikolas, Gautam Kamath, Daniel Kane, Jerry Li, Jacob Steinhardt, and Alistair Stew- art. Sever: A robust meta-algorithm for stochastic optimization. In International Conference on Machine Learning, pp. 1596-1606, 2019.
Yihe Dong, Samuel Hopkins, and Jerry Li. Quantum entropy scoring for fast robust mean estimation and improved outlier detection. Advances in Neural Information Processing Systems, pp. 6067- 6077, 2019.
Ian Goodfellow. Efficient per-example gradient computations. arXiv preprint arXiv:1510.0199) 2015.
Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. In Advances in neural information processing systems, pp. 8527-8531 2018.
Yifan Hu, Siqi Zhang, Xin Chen, and Niao He. Biased stochastic gradient descent for conditional stochastic optimization. arXiv preprint arXiv:2002. 10790, 2020.
Peter Huber. Robust estimation of a location parameter. In Breakthroughs in statistics, pp. 492- 518. Springer, 1992.
Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, and Li Fei-Fei. Mentornet: Learning data- driven curriculum for very deep neural networks on corrupted labels. In International Conference on Machine Learning. pp. 2304-2313, 2018
M Pawan Kumar, Benjamin Packer, and Daphne Koller. Self-paced learning for latent variable models. In Advances in neural information processing systems, pp. 1189-1197, 2010.
Kevin A Lai, Anup B Rao, and Santosh Vempala. Agnostic estimation of mean and covariance. In 2016 IEEE 57th Annual Symposium on Foundations of Computer Science (FOCS), pp. 665-674. IEEE, 2016.
Yuncheng Li, Jianchao Yang, Yale Song, Liangliang Cao, Jiebo Luo, and Li-Jia Li. Learning from noisy labels with distillation. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1910-1918, 2017.
10

---

Under review as a conference paper at ICLR 2021
Eran Malach and Shai Shalev-Shwartz. Decoupling" when to update" from" how update". In Advances in Neural Information Processing Systems, pp. 960-970, 2017.
Aditya Krishna Menon, Ankit Singh Rawat, Sashank J Reddi, and Sanjiv Kumar. Can gradient clipping mitigate label noise? In International Conference on Learning Representations, 2019.
Nagarajan Natarajan, Inderjit S Dhillon, Pradcep K Ravikumar, and Ambuj Tewari. Learning with noisy labels. In Advances in neural information processing systems, pp. 1196-12044 2013.
Jorge Nocedal, Annick Sartenaer, and Ciyou Zhu. On the behavior of the gradient norm in the steepest descent method. Computational Optimization and Applications, 22(():5-33, 2002.
Giorgio Patrini, Alessandro Rozza, Aditya Krishna Menon, Richard Nock, and Lizhen Qu. Making deep neural networks robust to label noise: A loss correction approach. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 1944-1952, 2017.
Adarsh Prasad, Arun Sai Suggala, Sivaraman Balakrishnan, and Pradeep Ravikumar. Robust esti- mation via robust gradient estimation. arXiv preprint arXiv: arXiv:/802.00 1802.0688.. 2018
Scott Reed, Honglak Lee, Dragomir Anguelov, Christian Szegedy, Dumitru Erhan, and Andrew Rabinovich. Training deep neural networks noisy labels with bootstrapping. arXiv preprint arXiv:1412.6596, 2014.
Mark Schmidt, Nicolas I. Roux, and Francis R Bach. Convergence rates of inexact proximal-gradient methods for convex optimization. In Advances in neural information processing systems, pp- 1458-1466, 2011.
Vatsal Shah, Xiaoxia Wu, and Sujay Sanghavi. Choosing the sample with lowest loss makes sgd robust. arXiv preprint arXiv:2001.03311 2020.
Yanyao Shen and Sujay Sanghavi. Learning with bad training data via iterative trimmed loss mini- mization. In International Conference on Machine Learning, pp. 5739-5748. PMLR, 2019.
John W Tukey. Mathematics and the picturing of data. In Proceedings of the International Congress of Mathematicians, Vancouver, 1975, volume 2. pp. 523-531, 1975.
Kun Yi and Jianxin Wu. Probabilistic end-to-end noise correction for learning with noisy labels. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7017- 7025, 2019.
Xingrui Yu, Bo Han, Jiangchao Yao, Gang Niu, Ivor W Tsang, and Masashi Sugiyama. How does disagreement help generalization against label corruption? arXiv preprint arXiv:1901004215, 2019.
Chiyuan Zhang, Samy Bengio, Moritz Hardt. Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. arXiv preprint arXiv: 161 1.05530, 2016.
Songzhu Zheng, Pengxiang Wu, Aman Goswami, Mayank Goswami, Dimitris Metaxas, and Chao Chen. Error-bounded correction of noisy labels. In International Conference on Machine Learn- ing. 2020.
A APPENDIX
A.1 CO-IGFILTER ALGORITHM
See algorithm 4
11

---

Under review as a conference paper at ICLR 2021
Algorithm 4: Co-PRI(L)
input: initialize wy and wg. learning rate y,. fixed т, epoch Tr, and Too.. iterations Near return model parameter wi and wy:
for T = 1, 1..,..., Tmax do
for N = 1, ..., Nmaz do
random sample minibatch M from Dz, D
(noisy dataset)
get the predicted label Y, and Y, from M by wy. Wg
calculate the individual loss I = L(Y, Y y), lg = L(Y, y))
Ølg
01g 8y.
calculate the gradient norm of loss layer score, = Bÿ, |l, scores = || l|.
sample R(T)% small--oss-layer--rrdiett---- update wy wy V my C(N wy), w/y wg V wy (NN g: wg)
instances by scorey and scoreg to get N,, N (selected dataset)
update model X Xa 7xp end
T T
Update R(T) = - min
end
Chen dake
Ascrdd dats
pr
Dorgerous moro fhtring
(a) When gradient filtering method failed pick (b) When loss filtering method failed to pick out out right corrupted data, the remaining coruppted right corrupted data, remaining corrupted data data is relatively smooth, thus has limited impact could be extremely sharp, thus has large impact overall loss surface.
overall loss surface.
Figure 1: Geometric illustration of the difference between loss filtering and gradient norm filtering. A.2 FURTHER ILLUSTRATION OF THE DIFFERENCE BETWEEN SPL AND PRL(G)
In this section, we will further illustrate the difference between SPL and PRL(G). In order to have more intuitive understanding of our algorithm, we could look at the Figure La and Tb Since are in the agnostic label corruption setting, is difficult to filtering out the correct corrupted data. We showed two situations when loss filtering failed and gradient filtering failed. As could that when loss filtering method failed, the remaining corrupted data could have large impact on the overall loss surface while when gradient filtering method failed, the remaining corrupted data only have limited impact on the overall loss surface, thus gaining robustness.
A.3 NETWORKS AND HYPERPARAMETERS
The hyperparameters are in Table For Classification, we use the same hyperparameters in Han et al. (2018). For CelebA, we use 3-layer fully connected network with 256 hidden nodes in hidden layer and leakly-relu as activation function. We also attached our code in supplementary materials.
Data' Hurerra
RaccSSoe
Leaming Ruse Optiniper
Momearur
CF-10
128
0.001
0.9
CF-100
128
0.001
0.9
ColcAA
512
0.0003
Adua
0.9
Table 4: Main Hyperparmeters
12

---

Under review as a conference paper at ICLR 2021
(a) 0.1 linady noise
0.2 linadv noise
(c) 0.3 linady noise
(d) 0.4 linadv noise
(e) c = 0.1 signflip noise
(f) = 0.2 signflip noise
(g) c = 0.3 signflip noise
(h) c 0.4 signflip noise
0.1 uninoise noise
(j) 0.2 uninoise noise
(k) 0.3 uninoise noise (l) 0.4 uninoise noise
(m) 0.1 mixture noise
(n)
0.2 mixture noise (0) c 0.3 mixture noise
(p) 0.4 mixture noise
(q) 0.1 pairtlip noise (r) 0.2 pairtlip noise
0.3 pairflip noise (t) 0.4 pairtlip noise
Figure 2: CelebA Testing Curve During Training. X axis represents the epoch number, Y axis represents the testing r-square. In some experiment, there is no cure for Standard and NormClip since they gave negative r-square, which will effect the plotting scale. The shadow represents the confidence interval, which is calculated across 5 random seed. As we see, PRL(G), PRL(L), and Co-PRL(L) are robust against different types of corruptions.
A.4 REGRESSION R2 ON TESTING DATA CURVE
The curve for CelebA data showed in Figure
A.5
CLASSIFICATION CURVE
The classification curve is in Figure
A.6 SENSITIVITY ANALYSIS
Since in real-world problems, it is hard to know that the ground-truth corruption rate, perform the sensitivity analysis in classification tasks to show the effect of The results are in Table As could see, the performance is stable if we overestimate the corruption rate, this is because only
13

---

Under review as a conference paper at ICLR 2021
333-
12
23
(a) CF1O with c = 0.3 symmetric (b) CF10 with c = 0.5 symmetric (c) CF10 with = 0.7 symmetric noise
noise
noise
16
(d) CF10 with 0.25 pairfip (c) CF10 with 0.35 pairflip (f) CF10 with 0.45 pairflip
noise
noise
noise
(g) CF100 with 0.3 symmet- (h) CF100 with 0.5 symmet- (i) CF100 with 0.7 symmel-
noise
ric noise
ric noise
18
(i) CF100 with e 0.25 pairflip (k) CF100 with 0.35 pairflip (I) CF100 with 0.45 pairflip noise
noise
noise
Figure 3: CIFARIO and CIFARIO0 Testing Curve During Training. X axis represents the epoch number, Y axis represents the testing accuracy. The shadow represents the confidence interval, which caleulated across random seed. we see, IGFilter(L.), and Co-IGFilter(.) are robust against different types of corruptions.
when we overestimate the could guarantee that the gradient norm of the remaining set is small. However, when we underestimate the corruption rate, in the worst case, there is no guarantee that the gradient norm of the remaining set is small. By using the empirical mean, individual gradient would ruin the gradient estimation, and according to the convergence analysis of biased gradient descent, the final solution could be very bad in terms of clean data. That explains why to underestimate the corruption rate gives bad results. Also, from Table 5 we could see that using the ground truth corruption rate will lead to small uncertainty.
one large bad
A.7 EMPIRICAL RESULTS ON RUNNING TIME
As we claimed in paper, the algorithm is not efficient. In here attached the execution time for one epoch for three different methods: Standard. PRL(G), PRL(L). For fair comparison, we replace all batch normalization module to group normalization for this comparison, since it is hard
14

---

Under review as a conference paper at ICLR 2021
Dasa
6- U.1
0.05
0.06
+ 0.1
65.07=0885
7047=0.67
75.7840.17
57.5620055
19:5-045
Sy1u-Syu-S0k SO%
09.21
72.55:0.455
75.45
0.09
7166540277
7..10-0.31
Cyi0-Sym- 70%
51.88=0644
5849=0097
60.36 0.2
60.89:041
=4.1100.6
02.60:0045
CF100-Sym-S0t
37.74-0441
39.7220.30 36
40.64
2011
45 030024000 36
43.92_0.61
CFT00 Sym 70%
24.40:0:47
25.50:0.45
27.27 1.000
20.00:0300
28.20 0,97
Table 5: sensitivity analysis for estimated
Method
Standard (Lower Bound))
PRU/C)
PREI)
CFI0-Pir-45%
37.05s
145.55
$4.80
Table 6: Execution Time of Single Epoch in CIFAR-I0 Data to calculate individual gradient when using batch normalization. For PRL(G), we use opacus libarary (https://opacuss.a ai/) calculate the individual gradient.
The results are showed in Table 6
A.8 PROOF OF CONVERGENCE OF BIASED SGD
We gave the proof of the theorem how biased gradient affect the final convergence of SGD, We introduce several assumptions and definition first:
Assumption (L-smoothness) The function o: Rd + R differentiable and there exists a con- stant L > 0 such that for all 01, O2 E Rdd we have (02) S
02 01) + 0 - 01 ||
Definition (Biased gradient oracle)
map g : R x D Rd. such that g( E) = Vo(0) + b(0,5) + ()) for bias b R Rd and zero-mean noise Rd D R. that is
(,n(0,g) () =
Compared to standard stochastic gradient oracle, the above definition introduces the bias term b. In noisy-label settings, the b is generated by the data with corrupted labels.
Assumption 3 (a-Bounded noise) There exists constants 0, v0 E gd
0, such that (e) s
Assumption 4 (G-Bounded bias) There exists constants C > 0, such that for any we have [b(0, <² V0 Rd
For simplicity, assume the learning rate is constant 7, then in every iteration, the biased SGD per- forms update 0,+1 +- 0, (.)(o,.)). Then the following theorem showed the gradient norm convergence with biased SGD.
Theorem (Convergence of Biased SGD(formal)) Under assumptions
define F
L.F
((Oo) 6² and step size min
T
(
oT
denote the desired accuracy as k, then
o²
T O
k2
iterations sufficient to obtain min E
iterations is sufficient to get minle(r) E V O) and performing more iterations does not improve the accuracy in terms of convergence.
Remark 2 Let l: = -
Since this is a standard results, similar results are showed in Bernstein et al. (001)) Devolder et al. (201)); [uu et al. (2020); Ajallocian & Stich (2020). we provide the proof here.
Proof: by L-smooth, we have:
((02) < 6(01) + (Vo(0)), 0z 01) + 102 0, ||
15

---

Under review as a conference paper at ICLR 2021
1 .. we have
by using 7 S
22L
Ed (01c+1) (0(0))) - 7 (V6 (gg)
LE
[gg Egrl 1l
+ E [gg 12)
2
= (010 - -1 (01)) V (01e) + bi)
E n 2 + E V (01.) 1 + ball?)
2
VV (01. + bell²) + 22L
(01) + 2/6 (-2 (11 11 Vo (01:) 10.1: + bb) +
E 1mg 11
2
2
(01.) 7 (- (0 + b,l + E ||
2
L
2
2
Since have ||b < <², |m|l 2 1 0², by plug in the learning rate constraint, have
Es (01+1)) 5 (01.) 7 (01,))1 + 2-2
0²
2
2 0²
Eø (0114) - o (61.) S - V 2Il + 5² Then, removing the gradient norm to left hand side, and sum it across different iterations, we could get
2
1
F T7
7Lo²
Σ
(0,0)
+ +
27
2
2
Take the minimum respect t and substitute the learning rate condition will directly get the results.
A.9 PROOF OF THEOREM 2
Denote G to be the set of corrupted minibatch, G to be the set of original clean minibatch and we have |G] = Gl = m. Let N to be the set of remaining data and according to our algorithm, the remaining data has the size N gradient, which is not discarded by algorithm [ to be the set of individual corrupted gradient, which not discarded. According to our definition, we have N individual good gradient, which is discarded, ARR to be the set of individual good gradient, which replaced by corrupted data. We have gradient, which is discarded by our algorithm. Denote the good gradient to be g: = a; W,, and the bad gradient be gi, according to our assumption, we have ||g: S
(1 c)m. Define A to be the set of individual clean
A B. AD to be the set of
A AD AR. BD is the set of individual corrupted
Now, we have the 12 norm error:
172
1
1
[l)(G) (NNII
g
gi
gi
m
EG
iEA
M
gi
g:
gi
n
m
71
SEA
JEB
N
Σ
71
Σ
T
1
Σ
Σ
0:
g:
+
g:
g:
+
g
m
11
m
11.
m
YEA
HEAD
jEAR
iEA
IEB
m
Σ
n
1
71
1
g8 +
g:
+
g:
m
m
n
m
72
YEA
iEAD
iEAR
iEB
M
ggi
71
g:
1
71
Σ
g
m
m
71
m
YEA
JEAD
SEAR
EEB
1
Σ 7
g:
8:
+
8:
m
m
M -
1
gill
||
+
g:
g
+
771
AD
AR
B
16

---

Under review as a conference paper at ICLR 2021
By using the filtering algorithm, we could guarantee that |
L.
Let
A
2,
have
B n2
n - x = =1 (1-c))) c) 2. AAR = m n = em, |A = m
A
AR = m I
mm
(1 (1- c)m xz Thus, we have:
1
1
1
[p(G) - ( S x
(n
x)
L + (m- n1
L + (n
x) --
mm
m
m
11
< x
m 11
1 -) L +
1
L + (m
11) -
1
L +
(n
x) -
nmm
m
m11
m
1 2E
JLL + -+ L ELL
zL( 2e 2
+2
To minimize the upper bound, we need be as small as possible since 2e 2 < 1. According to our problem setting, we have = me S (1 - 2 substitute back we have:
[l((G) (NNI < (1 22) Lame 2c -
+ 2T.
1 1 2
1 2L + 2L
4L 2L
Since e < 0.5, we use tylor expansion on
by ignoring the high-order terms, we have
[m(G) - (NN) = O(EL)
Note, the Lipschitz continuous assumption does not hold, then L should be dimension dependent.
A.10 PROOF OF RANDOMIZED FILTERING ALGORITHM
Lemma 3 (Gradient Estimation Error for Randomized Filtering) Given corrupted matrix G mgd generated in problem 2 Let G mxd be the original clean gradient matrix. Suppose are arbitrary select (1 c)m rows from G get remaining set N Raxd Let μ be the empirical mean function, assume the clean gradient before loss layer has bounded operator norm:
||||op & C, the maximum clean gradient in loss layer max; ||os|| k. the maximum corrupted gradient in loss layer max; || = v, assume < 0.5, then we have:
[l M(N) Ck 3E 422
+Cu 1-c
1-c
A.10.1 PROOF OF LEMMA 3
Denote G to be the set of corrupted minibatch, G to be the set of original clean minibatch and we have GG| G = m. Let N to be the set of remaining data and according to our algorithm, the remaining data has the size NN| gradient, which is not discarded by algorithm 3 to be the set of individual corrupted gradient, which is not discarded. According to our definition, we have N individual good gradient, which is discarded, AR to be the set of individual good gradient, which replaced by corrupted data. We have G = A AD U A.R. BD is the set of individual corrupted gradient, which is discarded by our algorithm. Denote the good gradient to be g; = a; W,, and the assumption, we have WWllop S C.
(1 c)m. Define A to be the set of individual clean
A.B. AD to be the set of
bad gradient be g. 6, W., according
17

---

Under review as a conference paper at ICLR 2021
Now, we have the 12 norm enror:
[l) (N)I
Σ
gi
Σ
gi
+
gi
72
EG
YEA
IEB
11
g
Σ
gi
Σ
Bi
iEA
:EB
1
Σ
Σ M
1
Σ
8:
8i
gi
g:
g
T1
711
11
T1
11
TEAD
iEAR
1
T1
Σ JEAD
11
1
Σ
71
1
Σ
0g
g:
g:
m
171
m
71
iEA
IEAR
1
11 m
g
1
11
gi
1
Σ
gill
Σ
(I)
m1
m
YEA
1EDD
SEAR
EE
Let A have
B 1- (1 1--)m--2 I, AR m- em,
AA
|A|
|A|||
|AR
m I - (m
x
(1 m x. Thus, we have:
(l((G) ((N)
Bi +
8
Σ
gill
+
m
m
n
AD
AR
B
Bill
81
gi
nu
m
m
7
AD
AR
B
For individual gradient, according to the label corruption gradient definition in problem assum-
ing the ||Wllop S C, we have ||g||
08.1A IA
||s
[all][Wlleee
Willop I Cllas]|- Also, denote max, |all = k
max; ||6:|| v, we have ||g:l| S Cki
Cv.
[|((G) (NNII S Cx m
n k + C(n - x) 1 C(m - -n) m C(n - x) -
Note the above upper bound holds for any x, thus, we would like to get the minimum of the upper bound respect x. Rearrange the term, we have
[[( [[N]] S Ca m n
bk Cn k + C(m nn) - k + C(n - x)) -
nm
1
2
1 )ack +Ck+ + Cu 1 Cre
C
m
n
k(2E 1)
2
Crx
CC + +Cv
m(1 6)
11
k(2c 1) -
Cr
+ Ck + Co
m(1 c)
k(2e 1) - m(1 c)
Since when 0.5,
0, we knew that x should be small possible continue
the bound. According to algorithm, we knew me m(1 me (1 22)m S x S r = (1 e) e)m. Then, substitute x = (1 - 2e),, we have
20 1
Cv 1- 2
[[( M(N)I S Ch(1 2c)
1-c +Ck+ Cv
1-c
3E 2
Ckk
+ Cr
1-6
1 -
18

---

Under review as a conference paper at ICLR 2021
A.11 PROOF OF THEOREM
According to algorithm3, we could guarantee that v S k, then, we will have:
[|((G) [N]]
& Ck 3E
4E²
Cu E
1
1-6
4c 4e2
Ck
1
4cCk
((E//)(CC is constant, k is the norm of q-dimensional vector)
A.12
COMPARISON BETWEEN SORTING LOSS LAYER GRADIENT NORM AND SORTING THE LOOSS VALUR
Assume we have d class label y Rd, where yk y: 0, We have two prediction Rd, E Rd.
Assume have d class label y E Rd where y4 1, V: + k. With little abuse of notation, suppose we have two prediction Rd q E RR Without loss of generality, we could assume that P1 has smaller cross entropy loss, which indicates р&: q
For MSE, assume we have opposite result
Ip
y/1
> l1q y-
Σ i
(1-
-
2 Σ
ch +
(2)
For each Pa; i + k; We have
Var(p;) Fll?)) F((p))
Σ
1
(1 px)
(d 1)2
(3)
izt
Then
Σ + (1- Σ ot + (1-qt)²
kgck
in
d
d
=Vars<((p;) (d 1)2
(1 px)² 2 Varifs(q + (d 1)2 p))
(1 -2
*Var<<((p)) Var (q:) 2 (d- 1)2 ((1 - q - -(1 -[1-pk)²
*Var Varizu 2 1)² ((px q2)(2 Pv - -qx))
19


Reviews:
[{"review_id": "kswogymT5UZ", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "# Summary\n\nThe papers studies the problem of robust machine learning, where the labels of the a fraction of samples are arbitrarily corrupted. The paper proposes an algorithm to tackle this problem and evaluates it on a standard datasets.  \n\n# Positives\n\nThe paper studies an important problem prevalent in modern machine learning, and proposes two algorithms to solve these problems. The experiments suggest that the proposed algorithm is better than the baselines.\n\n# Negatives \n\nThe paper does not cite highly relevant papers, overclaims its results, and the theoretical results in this paper are immediate. Moreover, the paper is not well-written. More details are given below:\n+ Page 1: \"Instead of developing an accurate criterion for detection corrupted samples, we adopt a novel perspective and focus on limiting the collective impact of corrupted samples during the learning process through robust mean estimation of gradients.\"\n+ This is not a novel perspective and has been known in robust machine learning community for some time [1,2]. These papers have the same underlying idea, but they are not discussed in this paper. [1] is only briefly mentioned in Remark 2, but the comparison is not fair. The results in [1] hold under fairly general conditions, where the results in this paper require the gradient to be uniformly bounded, which makes the problem significantly simple.\n+ Theorem 2 is a trivial result, well-known in field.  Moreover, the way it is presented is misleading and confusing. The error would depend on the quantile of  norms in G, which has been hidden under the O(.) notation. The proof is also missing from the paper. \n+ Assumption 1, i.e., Lipschitz continuity of the loss function is very restrictive, which is not satisfied by popular choices of loss function. This assumption trivializes the problem and restricts its applicability.\n+ In the same vein, Theorem 3 assumes unrealistic assumptions. The assumption that $||W||_{op} \\leq C$ is very restrictive and does not hold for usual learning tasks.\tThis assumption in a sense is restricting that the covariates x in $R^d$ have bounded norms, whereas the norm of a typical vector in $R^d$ increases as $\\sqrt{d}$.\n\n# Score\nI propose to reject this paper. Prior work ([1,2]) has studied this problem in a much greater generality, which are not discussed in this work. The assumptions in the present work are severely restrictive.\n## Other major comments:\n+ Robust linear regression, with arbitrary corruptions in responses, has been extensively studied in the literature but they have not been cited.  For example, see [3,4]. In particular, the least trimmed squares is an algorithm that removes outliers based on loss values, and comes with a theoretical guarantee via an alternating minimization algorithm [3,4].\n+ Theorem 1 is a folklore, and this should be reflected in main text. Currently, this information is only given in Appendix.\n+ The paper is not well written: \n 1.Proof of Theorem 2 is missing.\n 2. $O(.)$ notation hides the dependence on the important quantity in the papers.\n 3. Important notations have not been defined in the paper.\n 4. Abbreviations should not be used, for example, Thm., Algo., Asm., etc.\n 5. There are numerous typos and grammatical errors. For example, \"has a remarkably impact\". \n\n## Relevant papers\n1. Diakonikolas, I., G. Kamath, D. M. Kane, J. Li, J. Steinhardt, and A. Stewart. “Sever: A Robust Meta-Algorithm for Stochastic Optimization.” In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 97:1596–1606. Proceedings of Machine Learning Research. PMLR, 2019. http://proceedings.mlr.press/v97/diakonikolas19a.html.\n2. Prasad, A., A. S. Suggala, S. Balakrishnan, and P. Ravikumar. “Robust Estimation via Robust Gradient Estimation.” Journal of the Royal Statistical Society: Series B (Statistical Methodology) 82, no. 3 (July 2020): 601–27. https://doi.org/10.1111/rssb.12364.\n3. Bhatia, K., P. Jain, P. Kamalaruban, and P. Kar. “Consistent Robust Regression.” In Advances in Neural Information Processing Systems 30, NeurIPS 2017, 2110–2119, 2017. http://papers.nips.cc/paper/6806-consistent-robust-regression.\n4. Bhatia, K., P. Jain, and P. Kar. “Robust Regression via Hard Thresholding.” In Advances in Neural Information Processing Systems 28, NeurIPS 2015, 721–729, 2015. http://papers.nips.cc/paper/6010-robust-regression-via-hard-thresholding.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "ZO82cvdIh-M", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\n1. The related work section misses MANY related results on corrupted data and robust mean estimation. \n2. The related work section forget to mention existing theoretical results that apply robust mean estimation for robust gradient calculation. \n3. The related work section does not provide an accurate overview of existing results. For example, \"the algorithms themselves are NP-hard\" is not the correct statement -- NP-hard describes the hardness of a problem, not an algorithm. \n4. Collaborative learning methods seem to have no solid theoretical understanding and it is unclear why the proposed algorithm build on top of it. \n5. Regarding the novelty of the theorems: Theorem 1 studies convergence of biased gradient, which is another known research topic and has been studied before, but the authors have not discuss/compare their results with existing ones and the novelty may be overclaimed. Theorem 3 is for robustness guarantee with corruption only in the supervision, and existing results have shown O(\\epsilon) guarantee (for linear regression and its variants). \n6. I have not listed all the missing literature (I believe they are easy to find after a careful literature review), but I can add comment later if needed. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "TnBimQRFfj_", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors studied the problem of training neural networks under data poisoning, i.e., when a small fraction of the training data is corrupted by the adversary. They considered two data corruption settings, one allows both the data x and supervision y to be corrupted, which is called general corruption,  and one with only supervision y corrupted. Their first algorithm, which removes the datapoints whose gradient norm is large when computing the average gradient, applies to the general supervision setting. They showed their algorithm has eps\\sqrt(d) error or eps*L error, which can be quite large for high-dimensional and deep neural nets learning settings. Their second algorithm applies to the setting where only supervision y is corrupted, and the algorithm works by removing the datapoints whose output layer gradient is large. Assuming the clean data has bounded gradient, and the dimension of y is p, their algorithm achieves error eps*sqrt(p). \n\nWeakness: \n1.The authors claimed that compared to Diakonikolas 19, they improved the error from eps to sqrt(eps). However, the eps result relies on the fact that the gradient of good data has bounded norm, and I believe in that setting Diakonikolas 19 also achieves eps error. \n2. In paragraphs close to Lemma 1 and Lemma 3, the authors mentioned a randomized filtering algorithm, and proved Lemma 1 Lemma 3 for the algorithm. However, I can’t find the mentioned randomized filtering algorithm in the paper.\n3. Theorem 1 and Theorem 4 have no formal proof.\n4. Theorem 2 has no proof.\n5. In the experiment section, there is no comparison to other state-of-the-art algorithms, for example Diakonikolas 19.\n\nOverall, I think the theoretical result in the paper is incomplete, and the experimental evaluation is insufficient. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "N5EM1HwYPRs", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "pros \n1. The authors provide an insight that in noisy label learning, if the corrupted gradient is not far from the true one, then the learning algorithm could converge a sub-optimal result.\n2. Detailed experiments show the empirical evidence of the proposed algorithm over different kinds of label noise.\n\ncons \n1. The authors propose a method that only keeping the data with a small gradient norm in the training process to resist label noise. However, they do not verify that such a design is motivated by their theoretical results. Some important proofs for their key results are missing, e.g., Theorem 2 and Theorem 3, making this paper not self-contain. \n1. Many symbols are not well-defined mathematically. \n\n\nThis paper proposes a robust algorithm for noisy label learning. By keeping the data with a small gradient norm in the training process, the proposed algorithm could resist the label noise. Instead of making assumptions on the label corruption, the authors assume that the difference between the clean mini-batch gradient and the corrupted mini-batch gradient is bounded. Thus the proposed method could converge to the $\\epsilon$-optimal results. By dropping the data with a large gradient norm, the estimated gradient mean will not be far from the true one. The theoretical results make sense, but there lack detailed proofs to make this paper self-contain, e.g., for Theorem 2 and Theorem 3. The empirical studies on several datasets show the robustness of the proposed algorithm over different kinds of label noise.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The papers studies machine learning tasks in the presence of adversarially corrupt data (during training). In particular, it is assumed that the labels of a small constant fraction of the datapoints are arbitrarily corrupted.  The paper proposes a natural method to solve this problem and evaluates it on various datasets. As pointed out by the reviewers, the theoretical contributions of this paper are subsumed by a number of prior works (which were not initially cited). The experimental results of the paper are interesting. However, the method proposed  and evaluated is not particularly novel. In my opinion, the problems studied in this submission are important (in particular, the memory/space consideration in the context of robustness). However, this work still needs work and is not ready for publication.
Decision: Reject