Learning to Abstain From Uninformative Data
Authors: Yikai Zhang
Songzhu Zheng
Mina Dalirrooyfard
Pengxiang Wu
Anderson Schneider
Anant Raj
Yuriy Nevmyvaka
Chao Chen
Venue: Accepted by TMLR ()

Abstract: Learning and decision-making in domains with naturally high noise-to-signal ratios – such as Finance or Healthcare – is often challenging, while the stakes are very high. 
In this paper, we study the problem of learning and acting under a general noisy generative process. In this problem, the data distribution has a significant proportion of uninformative samples with high noise in the label, while part of the data contains useful information represented by low label noise. This dichotomy is present during both training and inference, which requires the proper handling of uninformative data during both training and testing. We propose a novel approach to learning under these conditions via a loss inspired by the selective learning theory. By minimizing this loss, the model is guaranteed to make a near-optimal decision by distinguishing informative data from uninformative data and making predictions.  We build upon the strength of our theoretical guarantees by describing an iterative algorithm, which jointly optimizes both a predictor and a selector, and evaluates its empirical performance in a variety of settings.

Learning to Abstain From Uninformative Data
Yikai Zhang Moryan Stanley
Yikai BonnauZhaannnnnggnsaannee....
Songzhu Zheng
Morgan Stanley
Mina Dalirrooyfard
Möna. www..rrwwwwoooddddooonnnnaaaaaaa
Morgan Stanley
Pengxiang Wu
prianguru@gmiil.com
Snap Inc.
Anderson Schneider
Antervon.Sharciserrrhnnanssssii com
Moryan Stanley
Anant Raj Morgan Stanley
Anent.Rajfmmorggaeetnllyy com
Yuriy Nevmyvaka Morgan Stanley
Yurig, Serrgyeeeemmuuuaa@urrraaaasaaaae
Chao Chen Stony Brook University
chao. chen. www..ngrroooooooooooooooooommoo. edu
Abstract
Learning and decision-making in domains with naturally high noisc-to-signal ratios such as Finance or Healthcare is often challenging, while the stakes are very high. In this paper, we study the problem of learning and acting under general noisy generative process. In this problem, the data distribution has a significant. proportion of uninformative samples with high noise the label, while part the data contains useful information represented by low label noise. This dichotomy present during both training and inference, which requires the proper handling of uninformative data during both training and testing. We propose novel approach learning under these conditions via loss inspired by the selective learning theory. By minimizing this loss, the model is guaranteed to make near-optimal decision by distinguishing informative data from uninformative data and making predictions. We build upon the strength of our theoretical guarantees by describing an iterative algorithm, which jointly optimizes both a predictor and a selector, and evaluates its empirical performance in a variety of settings.
1
Introduction
Despite the success of machine learning (ML) in computer vision (Krizhevsky et al. 2009 He al. 2016a Huang et al. 2017) and natural language processing (Vaswani et al. 2017 Devlin et al. 2018 the power of ML is yet to make similarly-weighty impact in other areas such as Finance or Public Health. One major challenge is the inherently high noise-to-signal ratio in certain domains. Tn financial statistical arbitrage, for instance, the spread between two assets usually modeled using Orstein-Uhlembent Uhlembeck processes [Oksendal
2003 Avellaneda Lee 2010). Spreads behave almost randomly near zero and are naturally unpredictable.
Equal contribution.

---

They become predictable in certain rare pockets/scennrioss for example, when the spread exceeds certain threshold, it will move towards zero with a high probability, making arbitrage profits possible. In cancer research, due to limited resources, only a small number of the most popular gene mutations are routinely tested for differential diagnosis and prognosis. However, because of the long tail distribution of mutation Trequencies across genes, these popular gene mutations can only capture a small proportion of the relevant. list of driver mutations of a patient (Reddy et al.] 2017). For a significant number of patients, the tested gene mutations may not be in the relevant list of driver mutations and their relationship w.r.t. the outcome may appear completely random. Identifying these patients automatically will justify additional gene mutation testing.
These high noise-to-signal ratio datasets pose new challenges for learning. New methods are required to ininformaatvve/lhh--oooee data in both the training and testing stages. The deal with the large fraction of uninformative/ hhigh-noise
source of uninformative data can be either due to the random nature of the data-generating process or due to the fact that the real causing factors are missing from the data. Direct application of standard supervised learning methods to such datasets is both challenging and unwarranted. Deep neural networks are even more affected by the presence of noise, due to their strong memorization power (Zhang et al. 2017); they are likely to overfit the noisc and make overly confident predictions where weak/no noo real structure exists.
In this paper, we propose a novel method for learning on datasets where a significant portion of content. has high noise. Instead of forcing the classifier to make predictions for every sample, we learn to first decide whether a datapoint is informative or not. Our idea is inspired by the classic selective prediction problem (Chow] [957). ; in which one learns to select subset of the data and only predict on that subset. However, the goal of selective prediction is very different from ours. A selective prediction method pursues balance between coverage (i.e. proportion of the data selected) and conditional accuracy the selected data, and does not explicitly model the underlying generative process. In particular, the aforementioned balance needs to be specified by a human expert, as opposed to being derived directly from the data. In our problem, we assume that uninformative data is an integral part of the underlying generative process and needs to be accounted for. By definition, no learning method, no matler how powerful, can successfully make predictions on uninformative data. Our goal therefore to identify these uninformative/ /high noise samples and, consequently, to train classifier that is less influenced by the noisy data.
Our method learns a selector, 9, to approximale the optimal indicator function of informative data, g". We assume that g" exists as part of the data generation process, but it is never revealed to us, even during training. Instead of direct supervision, we therefore must rely on mistakes made by the predictor in order to train the selector. To achieve this goal, we propose a novel selector loss enforcing that (1) the selected data best fits the predictor, and (2) in the portion of the data where we abstain from forecasting, the predictor's 's performance is similar to random chance. The resulting loss function is quite different from the loss in classic selective prediction, which penalizes all unselected data equally.
We theoretically analyze our method under general noisy data generation process, imposing an additional structure on the standard data-dependent label noise model (Massart & Néddlec 2006 Hanneke 2009). We distinguish informative versus uninformative data via a gap in the label noise ratio. A major contribution of this paper is the derivation of theoretical guarantees for the selector loss using empirical risk minimization (ERM). A minimax-optimal sample complexity bound for approximating the optimal selector is provided. We show that optimizing the selector loss can recover nearly all the informative data in a PAC fashion (Valiant, 1984). This guarantee holds even in challenging setting where the uninformative data has purely random labels and dominates the training set. By leveraging the estimated selector, one can further pick out the informative subset of the training samples. We prove that the classifier generated through risk minimization conditional on the informative subset exhibits a superior upper bound on risk compared to a conventional classifier trained using the complete training dataset.
The thcoretical results generalize the method to a more realistic setting where the sample size is limited. Furthermore, the initial predictor is not sufficiently close to the ground truth. Our method yields an iterative algorithm, in which both the predictor and the selector progressively optimized. The selector is improved by optimizing our novel selector loss. Meanwhile, the predictor is strengthened by optimizing the empirical risk: re-weighted based on the output from the selector, where uninformative samples identified by the
2

---

selector are down-weighed. Experiments on both synthetic and real-world datasets demonstrate the merit of our method compared to existing baselines.
2 Related work
Learning with untrusted data aims to recover the ground truth model from a partially corrupted dataset.
Different noise models have been studied, including random label noise (Bylander 1994 Natarajan et al.
2013), Massart Noise (Massart & Néddlec 2006; Awasthi et al.) 2015: Hanneke 2009 Hanneke & Yang 2015
Yan & Zhang, 2017
Diakonikolas et al.] 2019 2020; Zhang & Li 2021 and adversarial noise (Kearns & Lip
1993 Kearns al.
1994 Kalai et al., 2008 Klivans et al. 2009 Awasthi et al. 2017). Our noise model is
similar to General Massart Noise (Massart & Nódêlec 2006 Hanneke 2009 Diakonikolas et al. 2019), where
the label noise is dula-dependenl, and the label can be generated via purely random coin flipping. The major distinction in our noisy generative model is the existence of some uninformative data with high noise in label compared to informative data with low noise in label. We characterize such uninformative mninfornative/inooraatte /nnfommtiie data structure via a non- vanishing label noise ratio gap. While there exists a long history of literature studying training classifiers with noisy label (Thulasidasan et al. 2019), we are the first to investigate learning a model robust against label noise at inference stage. We study the case where uninformative samples are an integral part of the generative process and thus will appear during inference stage as well, where they must. be discarded if detected. We view this as realistic setup in industries like Finance and Healtheare.
Sclective lcarning is an active rescarch area (Chow 1957:
1970 El- Yaniv et al. 2010 Kalai et al. 2012
Nan & Saligrama 2017 Ni et al. 2019 Acar et al. 2020 Gangrade et. al. (021a) that expands on the classic
selective prediction problem. It focuses on how to select a subset of data for different learning tasks, and has
also been further generalized to other problems, c.g., learning to defer human expert (Madras et al. 2018
Mozamnar & Sontag 2020). We can approximately classiff existing methods into categories: Monte Carlo
sampling based methods [Gal & Ghahramani] 2016 Kendall & Gal 2017 Pearce et al. 2020), margin based
methods Fuunera & Roli 2002: Bartlett & Wegkamp/ 2008 Grandvalet et al. 2008 Wegkamp et al. 2011
Zhang et al. 2018). confidence based methods (Wiener & El- Yaniv 2011] Geifman & El- Yaniv! 2017
Jiang
et al. 2018) and customized selective loss (Cortes et al.J 2016 Geilman & El-Yaniv 2019: Liu et al.
2019:
Gangrade et al. 2021c). Notably, several works propose customized losses, and incorporate them into neural networks. In Geifman & EI- Yaniv 2019), the network maintains an extra output neuron to indicate rejection of datapoints. Liu et al. (2019) introduces the Gambler loss where cost term is associated with each output neuron and doubling-rate-like loss function is used to balance rejections and predictions. [hhllaidaaaa et al. (2019) also applies extra output neuron for identifying noise label to improve the robustness in learning. Huang et al. (2020) adopts a progressive label smoothing method which prevents DNN from overfitting and improves selective risk when applied to selective classification task. Cortes et al. (2016) performs data selection with an extra model and introduces a selective loss that helps to maximize the coverage ratio, thus trading off a small fraction of data for better precision. Sharing a similar spirit with (Kalai el al., ) (Gangrade et al., (021c) applies a one-sided prediction method to model a high confidence region for each individual class, and maximizes coverage while maintaining a low risk level.
Existing works on selective prediction are all molivated by the trade off between accuracy and coverage i.e. one wants to make confident predictions to achieve higher precision while maintaining reasonable recall. To the best of our knowledge, our paper is the first to investigate the case where some (or even the majority) of the data is uninformative, and thus must be discarded at test time. Unlike the seleetive prediction, our framework considers a latent neverlee ground truth indicator function about whether a data point should be selected or not. Our method is guarantcod to identify those uninformative samples.
3 Problem formulation
In this section, we describe the framework for the inherently noisy data generation process that we study. Definition 1 (Noisy Generative Process). Let a E (0, 1) be a moblem-dependent constant and 2p S Rd be the support of D., below. We define Noisy Generative Process by the following nolation ~ D., where
3

---

x Du
with prob. a (Uninformmative) (Informative).
Da =
x D1
with prob.
Given S R let the ground truth labeling function f X + {0,1} J in hypothesis class F. Suppose {Qg, }1r} partition of S2p. Let M(x) E (A, 2) with X > O, the latent
status
E {1,0} has posterior distribulion:
N(x),
i E Qv if x E Ωr.
P[ 1 + X(r),
(2)
The observed data (x, v) is generated according to:
x ~ Dai
2 ~ P P[z|a]]
(3)
y (ernoul
if = === 1.
y y= fx),,
Since A(x) > 0, x from Dy has a lower probability to be observed with true label compared to Q2 thus it can viewed as uninformative data in a relative sense. On the contrary, x from Q2 can be viewed as informative data. Our Noisy Generative Process follows standard data-dependent label noise, e.g., Massart Noise (Massart
& Nédélec 2006) and Benign Label Noise (Hanneke) 2009 Hanneke & Yang 2015; Diakonikolas
al,
2019) with label noise ratio (2g" (=)==) Indeed, one can always choose X(x) E [0, E) and a E [0,
2
to replicate General Massart noise. Compared to classical label noise models, the assumption X(x) X introduces a label noise ratio gap, which distinguishes the informative and uninformative data. In Equation 3
the (ernoulli(0.5) label noise serves as a proxy for "white noise" in label corruption. When A(x) = and x E Luy, Bernoulli(O.5) random label noise can be viewed as the strongest known non-adversarial label noise, of both theoretical and practical interest (Diakonikolas et al. 2019). Such Bernoull((0.5) random label noise could happen when hard-to-classic examples are shown human annotators (Klebanov Beigman 2010),
when fluctuations in financial markets closely resemble random walk (Taa 2005).
A typical setting that is studied in this work is the case that both values of - and A(x) are non-vanishing, ie., there is significant fraction of uninformative data (large a) and the label noise ratio gap is distingusshable between informative and uninformative data (large X).
The next definition describes a recoverable condition of the optimal function for the latent informa- atre/uninformative status z.
Definition 2 (G-realizable). Given support Dep and Max) (X, let the posterior distribution of be defined in Equation (2). We say Up is G-realizable if there exists g² E G X {0,1} satisfying g²(x) L[P[2 1|x] 7
Ideally, one would want to select all informative samples where signal dominates noise. This can be done via recovering (-),,, which we view as the ground truth selector we wish to recover. The G-realizable condition is analogous to the realizability condition (Massart & Nédélec 2006 Hanneke & Yang 2015) in the classical label noise problem. The major difference and challenge in recovering g g"(-) compared to learning a classifier, is that there is no direct observation on the informative wwwttiivnnoonnnoonmneee this work is proposing a natural selector risk which recovers g" (-) without observing the latent variable 2.
status z. The major contribution of
Having introduced the data generation process, we now describe the learning task: Assumption 1. Data Sn = {xt, V 1 i.i.d generated according to the Noisy Generative Process (Defini- tion 7), with f E F and support 22p salisfies G-realizable condition.
Given the above assumption, we are interested in the following learning task: Problem (Abstain from Uninformative Data). Under Assumption | with i,i,d observations from Da, we adm to learn a selector ğ E G that is close to g" (x) and predictor f with low selective risk.
4

---

Ground Tuth y-O Ground Trath yo1
Gutpet froil Dlsatier Gulput of
Seledoo: Infermtive
Selector: Unintormtive
informaive Data Unintomat Data
Clsss Per Correct Clsss incomett
Prediction 0
Selector: Acstsined
(a) Inform/ Unniforrnio Data
(b) Correct/Incorrect /Icoorecct classification
(c) Selective classification
Figure Illustration of the learning strategy when x distributed according a Gaussian mixture. We replace the 0-1 loss with hinge loss and train SVM models for f and g. (a) upper panel shows the original dataset and bottom panel shows the region of informative (easy) and uninformative (hard) data. (b) shows that the classifier has high accuracy in the informative region, but low accuracy in the uninformative region. In (c), the selector trained with f successfully recovers informative support thus resulting in low selective risk, and we abstain from making a prediction elsewhere.
4 Our method
In this section, we present our approach to learning and abstaining the presence of uninformative data (Problem D. The main challenge that the latent informative/uninformnni unknown. Our main idea to introduce novel yet natural selector loss function that trains a selector based on the performance of the best predictor (Section 4.Ih. In Section 5 we present our main theoretical result. We show that, given any reasonably good classifier, by finding selector minimizing the proposed selector loss, solve Problem with minimax-optimal sample complexily. Inspired by the theorelical results, in we propose practical algorithm that iteratively optimizes the predictor and the selector.
status of a datapoint is
Section i
4.1 Selector loss
In an idcalized setting, when access to latent informative/ uninformaaiiie v variables {(0,, 2)) is available, recovering 9" shares similar spirit with learning classifier under label noise. It suffices to minimize the following classical classification risk
Non-Realizable Riskig; S.)) Σ H(g(x)) + 2
(4)
However, in practice z is never revealed. To learn a selector without direct supervision, we have leverage the performance of a predictor f. We propose to replace in the Equation 4 with psenuo-informmaive 1 (f(x)) = v), which has randomness coming from and noisy label y.
formalive label
Definition 3 (Selector Loss). Let scalar 3 > 0 be some weight coefficient, given f E F and its selector G, we define the following empirical version of weighted 0-1 type risk w.r.t g(.) as selector risk:
Rs, (g. J,3) = Σ { (1)(()()))) > 0) +) = y1 11 s o3}
(5)
where S, = ( 1 defined Assumption
The selector loss is also natural metric evaluate the quality of the selector. This loss penalizes when (1) the predictor makes correct prediction datapoint that the selector considers uninformative and

---

abstains from, or (2) the predictor makes an incorrect prediction when the selector considers informative. Intuitively, the loss will drive the selector to partition the domain into informative and uninformative regions. Within the informative region, the predictor is supposed to fit the data well, and should be more accurate. Meanwhile, within the uninformative region, the label is random and the predictor is prone to make mistakes.
Note that there are two types of errors penalized in the selector loss: an incorrect prediction on a selected datapoint, (f(x) + y))(g())= > 0), and a correct prediction on an unselected datapoint, (f(x) = y)A(g()) - 0). Since the label noise is non-adversarial, y tends to have higher probability of coincidence with f*)),, introducing imbalance on the peeudo-iformatiiee label. We thus use 3 to weigh these two types of errors in the loss. An analysis can be found in section A.1 on the choice of 8. Our theoretical analysis suggests that, for a wide range of 3, the accuracy of the selector is guaranteed. In particular, the range of 3 depends on the value of label noise ratio gap X, empirically, many values work - see Appendix and we choose 3 = 3 throughout. Our experiments show stability with regard to this choice.
Learning selector with the novel loss. To learn selector, one can follow standard procedures e.g., empirical risk minimization (ERM), to get a predictor f with reasonable quality. The selector can be estimated by minimizing the selector loss 9 = arg min R.s. (g. f.)), conditioned on the estimated prodictor f.
In Figure 1 we show an example of using the FRM strategy using SYM with 0-1 loss replaced by hinge loss. In this case, the losses are all convex and the empirical minimizers f and ğ can be computed exactly.
In practice, however, empirical minimization is not always possible, as optimization for general complex models (e.g., DNNs), resulting in non-convex losses, remains an open problem. We therefore propose a practical algorithm in the spirit of our theoretical results: it jointly learns f and g by minimizing the selector loss and a reweighed classification risk iteratively (see Section G
5 Theoretical results
In this section, we present our theoretical results. The main one can be summarizod in the following (informal) statement.
Main Result (Informal) For any reasonably good predictor F, with sufficient data, the selector î estimated using I arg mingeg Rs, (g. f. ()3 is sufficiently close to the targets g" with high probability, where Sn = [(a;, f defined in Assumption 1 Furthermore, training predictor f only using informative data selected by ÿ improves selective risk.
Remark 1. The loolkit we use in the proof is a bernstein-type inequality for fast generalization rate under margin condition (Massart & Nédélec, 2006- Van Erven et al, 2015: Li & Lid, 2021). We also provide on information theoretic lower bound construction in section A. to show our selector risk bound is minimax- optimal. Our construction of the lower bound is motivated from (Ehrenfeucht et al., 1989: Blumer al., 1989) and Le Cam's 's method YYul 1997). Due to space constraints, detailed proofs of our theorems are provided in the Appendia. To avoid lengthy technical definitions, we only present a light version of our results in the format of finite hypothesis class. The extension to VC-class using Local Rademacher Average tools [Bartletf
et al. 2005) is shown in section B
Theorem 1 Minimax-Opitmma Selector Risk Bound). Let S, = {xx, 1:) be i.i.d sample from Data
Generative Process described in Definition under Assumption 1 with f(()) E and g²()) E 9, |F| < 0o
9 < 00. Given X, let 3 E
[3-2A + X, min 3421 __ L142A
1-22 1-4x2 10)] For any f(.) E F let g = arg min Rs. (g: f. f). gEg
Then for any E > 0, 8 > 0 such that the following holds: For n 2 max 320² logi 10 that salisfies one of the following condition:
243 213
}. and for f
For any f(-) F such that Ex ff(x) + f*(x)] < 8 with prob at least 1 - 8,
For any T(- E F such that E.n (f()) # fz) l E 2i) S sia with proh at least - 6,
6

---

The following holds with probability at least - -25:
R(T: .,,)),))))) 33 f S
Remark 2. The assumption that Fe (fxx) + (x)) Se could be achieved via an FRM on classifiention (()()))= yi} under some margin condtions [Bousqued] 2004 Massart & Nédélec 2006; Bartleti et al. 2005). In practice, one con also apply some methods beyond ERM to obloin I Namkoong & Duchi
loss
2017). In particular, in the case X = 2, the data in support ay is un-learnable as y are purely random. While upprozsmating f on the full support is not possible in general, one can control the conditional risk Ex (x(f(x) (x)]x S2yl via a standord ERM schema (see proof in appendix Section A.7). We stress that Theorem holds for any classifter that is close to f", even the case where f and 9 are trained on the same dalasel.
Corollary 1 (Recovering 9¹). Given conditions in Theorem 7 is we choose 3 = 3, we have:
gg < 4e + 2A)
Em [x(1)((ar) + (x)))
(6)
Corollary [ suggests that by minimizing the empirical version the loss from Definitions 3 one can recover g' in PAC fashion. The thcoretical guarantee holds even under a very challenging case where a > 0.5 and == -; .egg, the majority of the data has purely random labels. The analysis of the selector loss (Theorem relics on the quality of the classifier f. But since we know that 9 is able to abstain from uninformative data, we can retrain I beyond standard ERM, with upweighted informative data, therefore improving the accuracy of f. Such circular logic naturally leads to a alternating risk minimization schema. We formulate such schema in equations
( and g Next theorem studies the risk bounds of classifiers by minimizing the empirical risk conditional on the samples selected by J. The risk bound improves the conditional risk compared to the standard empirical risk minimization method up to some problen-dependent constant.
Theorem 2 (Joint Risk Bounds ). Let S, ( vs) 1i be i.i.d samples from the Data Generative Process described in Definition 1 under Assumption 1 with (-) E F and g²()) E G, |Fl < 20, 191 < co and
log( 101)
log LE
A(x) < h,h 0. For any e > and 1, if
64
and:
X
R
1 n
ERM Classifier: f= arg min fEr n
Σ 1(f(z.) # y1
(7)
i=1
ERM Selector: " arg min Rs, (0: 8,3)
(8)
gEg
1
Subset-ERM Classifier: J = arg min fEF 1=1
1 (0(m;) ()1((()))) # m.
(9)
the following inequalities hold with probability at least 1 - 35:
[z ( (x)) S E. [= g" (=) $ 4=
(10)
(11)
fx # (g) ' (z) 1| S h
(12)
Theorem 2 implies the convergence of the alternating minimization procedure between equation by setting f := f, due to the fact that equation
g and
(2) satisfies the pre-requisite condition on f in Theorem
Another take away of Theorem 2 is an improved selective risk of classifier resulting from training only with
samples picked out by the selector. To see this, we note that Massart Noise condition is satisfied with margin
& under the assumption that X(x) < { -h. The risk bound under standard margin condition in equation (10
2
follows from a standard result (see Section 5.2 in (Bousquet) 2204)). Indeed, equation (10 implies the
following risk bounds conditional on Ωy
En(j(x) + (x)))) =1] 1]
(13)
7

---

Which is improved by problem-dependent constant factor, of order h², compared to equation (2 While risk bound in equation (10) is minimax optimal in general due to (Massart & Nédélec 2006), the problem- dependent structure introduced in Definition alloww tighter risk bounds on sub-regions. The refined selective risk could be achieved by a weighted-ERi ERRM, with binary weights {0,1} given by selectors.
Remark 3. The analysis presented in [Cortes et al., 2016) suggests that the generalization bound ezhibits rate of O(L/vn) This risk bound covers scenarios where there is presumption of ground truth model, nor any gap on the label noise ratio to distinguish informative and uninformative data. In comparison, the risk bounds eslablished in Theorem indicale a Jaster, mini-mar oplimal rale of (1//n) by capitalizing on the structural characteristics of the underlying data generation process, This contrast in assumptions stems from the motivation behind selective learning. While in the works of (Cortes et al., 2016 Geifman & El- Yaniu, 2019), the selective loss is devised with focus on coverage ratio, i.e., optimizing for higher precision (selective loss) by trading coverage ratio, our opproach is tailored to distinguish dala that is inherently unlearnable and unpredictable. This discrepancy leads to alternative theoretical outcome. The analysis in (Cortes et al. 2016) primarily centers on selective risk, whereas our theoretical analysis places emphasis on the selector's 's capabilily to differentiate belween informalive and uninformativee data without adjusting the rejection cost imposed by a human.
Algorithm 1 Iterative Soft Abstain (ISA)
1: Input: Dataset Sn (II1,,1),,... (In, Bo))}, weight parameter: random initial classifier fo and selector 8". number of iterations T 2: for <- 1,1,....
T 1,..,, do
3: Optimize loss to update predictor f Em= (2: log(f" + (1- - ys) log(1 (x:))) Approximate the aseudo-informative label', 2 = (1 > }}} =
Optimize loss to update selector g': Ein {zf log(0' (æ:)) 3(1 log(1 (xx))))
6: end for
7: Output: fT,0T
6 A practical algorithm
Motivated by our theoretical analysis, we propose a practical algorithm that shares a spirit similar to the selector loss. From a computational standpoint, we replace the binary loss by cross-entropy loss instead and require that both f and g have continuous-valued output, ranging between 0 and 1 instead of binary output. The label y also needs to be processed so that the values are in the {0, 1} range.
Following alternating steps given in equation S and 19 the practical algorithm trains both predictor and selector in an itcrative manner. To accelerate the training, instead of applying an alternating minimization schema, we relax the requirement for minimization oracles by stochastic gradient updates. During the joint optimization process, the predictor is counting on the selector to upweight informative data. By putting more effort on the informative data, we wish to improve the performance of the predictor on informative data, in equation I2 Algorithm I shows the logic above. A pictorial example of Algorithm I performance can be found in Figure in the Appendix.
7 Experiments
In this section, we test. the efficacy of our practical algorithm (Algorithm 1 on both publicly-available and semi-synthetic datasets. The code for reproducing the results could be found in https://gitthbb com/ following questions:
tn The empirical study aims to answer the
Q1: How does Algorithm compore to baselines on semi-synthetic datasets in recovering ground truth selector g"
8

---

The results are presented in Figure 2 the ability of each baseline in recovering the o given different values for the noise ratio gap À and sample size constraints. One can see our proposed method outperforms all listed baselines in this particular task.
3 and 4 We conducted semi-syuthetic experiments to evaluate
Q2: Does our algorithm oulperform ERM on selected data points? ?
One important motivation behind selective learning is to find a model that outperforms the ERM on the informative part of the data. This is challenging task as ERM is minimax optimal in general
sotting(Massart & Nédélec. 2006 Bartlett & Mendelson 2006). We show in Table | that our proposed algorithm does exploit additional problem structure and achieves lower risk on selected datapoints.
Qa: How does Algorillm work on real world dalasels compared lo selective learning baselines?
On real world datasets, Algorithm 1 consistently shows competitive/superrtt baselines in the low coverage regime, e.g., the proportion of data chosen by the selector being below 20%. These empirical results suggest that that our method is good at picking out strongly informative data. This observation supports theorem 2
performance against other
Baselines. We compare our method to recently proposed selective learning algorithms. (1) SelectiveNet [Geifman & El- Yaniv 2019). which integrates an extra neuron as a data selector in the output layer and also introduces a loss term to control the coverage ratio; (2) DeepGambler (Liu et al. 2019), which also maintains an extra neuron for abstention and uses a doubling-rate-like loss term (i.c., gambler loss) to train the model. (3) Adaptive (Huang et al. 2020) uses the moving average confidence of classifier as the soft-label for the training and an extra output neuron is used to indicate abstention; (4) Oneside (Gangrade al., 2021b) formulates selective classification problems as multiple one-sided prediction problems where each data class's coverage is maximized under error constrains. The implemented classifier is trained to optimize a relaxed min-max loss. The selection then is performed according to the classifier's confidence; (5) We also create a heuristic baseline that selects data using the model prediction confidence, which we refer to as Confidence. The intuition behind this heuristic bascline that informative data should have higher confidence compared to uninformative data.
Our Algorithm: ISA and ISA-V2. Several specific implementation details need to be mentioned. First of all, we empirically get better performance if we average 2' (in Algorithm 1 over past 10 epochs instead of using z' from the most recent epoch. Second, we use a rolling window average of past 10 epochs of g for sample weights instead of only using current epoch's 's output. Furthermore, inspired by existing literature, we empirically get better performance in real-world datasets we allow both the classifier model and the selector model to use the same backbone network share data representations. We add an extra output neuron to the classifier model to achieve this goal. Finally, we use focal loss Lin et al. (2017) in order to deal with imbalance of informative versus uninformative data. The selector focal loss has the following format:
l(2),0)) Σ [- - 9:) log + 51 - -2,70
log (1 -9:)))
(14)
where z; = -1f = 31 is the classifier i's correctness on ith input. We use fJ as the abbreviation for the ith entry in (xx))).
We will denote as ISA the modified version of Algorithm 1 that adopts the rolling average and focal loss, and ISA-V2 as the version that adopts all 4 relaxations. Notice that both ISA and ISA-V2 can be easily applied in multi-class scenarios in practice.
Experiment Details. We use lightweight CNN for MNIST+Fashion. Its architecture is given in Table 5 in section D.1 We use ResNet18 (He et al. 2016b: for SVHN. For all of our synthetic experiments, we use Adam optimizer with learning rate 1e-3 and weight decay rate 1e-4. We use batch size 256 and train 60 epochs for MNIST Fashion and 120 epochs for SVHN. The learning rate is reduced by 0.5 at epochs 15, 35 and 55 for Fashion and is reduced by 0.5 at epochs 40, 60 and 80 for SVHN. We repeat each experiment 3 times using seeds 80, 81 and 82.
9

---

Regarding hyper-parameters for each baseline, mainly follow the setting given by the original paper. Except for DeepGambler in Fasiin,, which the original paper does not contemplate. We set the pay-off 0 = 1.5 instead of 2.6, which gives better performance for DeepGambler. For Oneside, since we are not maximizing the coverage (the optimal coverage is fixed in our problem), we fix the Lagrangian multiplier I to be 0.5 and use as the score to compute average precision. We made our best effort to implement this algorithm, given that the authors have not released the original code. We list all the hyper-parameters Table G in section D.I
We conduct all our experiments using Pytorch 3.10 (Paskke et al, 2019 We execute program Red Hat. Enterprise Limux Server 7.9 (Maipo) and use NVIDIA V100 GPU with cuda version 12.1.
7.1 Experiments Using Semi-Synthetic Data for Q1
Dataset Construction: We explicitly control the support of Coformativeeuninformmenn FaSIi--IFSh dataset, images from MNIST are defined to be uninformative, while images from Fashion-MNIST are set to be informative. For SVHN(Netzer et al., 2011) dataset, class 5-9 are set to be uninformative and class 0-4 are set to be informative. Datascts are constructed with different values of informative data fraction a and label noise ratio gap À driving the noisy generalive process. We inject label noise accordingly to Definition [ by setting {(x) = x. Note that when we vary a, we only constrain the sample size the training sct, while the testing set we always use the complete testing set for performance evaluation.
data. For
We present two sets of experiments. In the first experiment, we set À = 0.5, which gives realizable informative data and completely randomly shuffled uninformative data. We fix the number of uninformative data points and increase the number of informative ones as a proxy for different value of a in the Definition [ Specifically, we run our experiments with ilaformative Data #Uninformative Data Ita E (1.0, 0.75, 0.5, 0.25). Furthermore, we conduct our
a
experiments in both a complete dataset setting and a sparse dataset setting to test each algorithm's sample efficiency. In a complete dataset setting, we use 100% of all informative data and uninformative data, whie in a sparse dataset setting, we use only 25% of samples.
In the second set of experiments, injoct 10%, 20% and 30% proportion of uniform label noise into the informative part and also inject 80%, 70% and 60% uniform label noise into the uninformative part.
Evaluation Metrics. Every baseline will generate score besides their classification output to decide if the input should be abstained. For SLNet, DeepGambler, Adaptive and ISA-V2, the score is given by the extra neuron. For Confidence and Oneside, such score is its maximum probabilistic oulput. For ISA, the score is given by the selector model. In the semi-syuthetic experiment, we can calculate the average precision (AP) using the selection score and ground-truth informative versus uninformative binary label to test each baseline's ability in recovering g". In study, especially focus the case where is small and informative data is the minority part of the dataset. In this situation, there is severe class imbalance in the latent informative/ tameeemninfrrmaiive labeling. AP is preferred over other metries, like Fl or AUC, in this scenario (Saito & Rehmsmeier) 2015). In Table 2 Table g and Table we use transformation - log (1 x) to make the performance difference visually distinguishable.
Results and Discussion. There are empirical observations we can get from Figure 2 3 where we manifest the performance difference though the transformation -log(1 - Average Precision) for differentiation purpose. More detailed results can be found in appendix section D.2. Firstly, the proposed ISA method outperforms all basclines under most of the scenarios according to the average precision criterion, which supports the superiority of the proposed method in recovery g". Secondly, we observe that our method's performance is more stable and suffers less deterioration as the data size becomes more limited or the label noise becomes stronger. Finally, many cases, Confidence, which simply uses the confidence of a model trained with vanilla cross entropy loss, is a competitive baseline. Our proposed method outperforms Confidence in most scenarios.
10

---

Figure 2: Average Precision (AP) v.8 Different a. Numerical results in Table
MNIST
8-
Jogl2--A
2
1
d=0.20
d=0.33
0=0.43
d=0.50
e=0.20
d=0.33
a=0.43
d=0.50
confidence
selectivenet
deepgambler
adaptive
oneside
isa
isav2
Figure 3: Average Precision (AP) v.s Different 25% Samples Size. Numerical results in Table
MNIST
SVHN
10
8-
Hog(1-AP7
2-
2-0.20
0-0.33
2-0.43
0-0.50
G-0,20
d=0.33
G-0.43
0
confidence
selectivenet
deepgambler
adaptive
oneside
isav2
Figure 4: Average Precision (AP) v.s. Different X
Numerical results in Table
MNIST
SVHN
10
(og(1-AP)
0=0.66
1=0.17
A=0.28
0=0.06
A=0.17
A=
confidence
selectivenet
deepgambler
adaptive
oneside
isav2
11

---

7.2 Experiments Using Semi- Synihhynt Data for Q2
Evaluation Metrics: The construction of inforrattieuuunnfoontivvv data follows Section 7.1 In this section, want to mcasure the classification performance of the proposed method conditioning on selected data points, by looking at the selective risk (SR) metric. Its formal definition is given by SR, = Σ, [arg max f(x,) # (ū(x;) > 4)}, where the quantity q is used to control the coverage. The confidence method could be viewed as a proxy for ERM method with following adjustments: 1) replacing cross-entropy as surrogate loss for binary classification loss 2) replacing selector 1 (0(z) 2 d} by using estimated margin based selection rule 1 [(fERas(()) - 1/2] 2 q). Coverage is defined as the proportion of selected data to the whole dataset. In this experiment, we always calculate SR at the ground-truth coverage = 0.5. It is the proportion of informative data that we mixed into the lesting set. Ideally, we hope that the selector can select all but only informative data, which should give the lowest risk given the coverage level.
uninformative
Table 1: SR 4. ERM v.s ISA
100% Sample
25% Sample
Dataset
Method
72 = 0.3
T = 0.2
T = 0.1
T3 0.3
T = 0.2
7: = 0.1
myy = 0.6
TV 0.7
Tr = 0.8
Ty = 0.6
TV = 0.7
Ty = 0.8
Confidence
0.387 + 0.001
0.293 + 0.005
0,188 = 0.005
0.469 = 0.002
0.362 + 0.002
0.235 + 0.002
FashT+
ISA
0.376 + 0.004
0.280 + 0.004
0.186 + 0.003
0.446 # 0.002
0.338 + 0.005
0.233 +: 0.004
ISA-V2
0.378 + 0,008
0,281 + 0.001
0.192 =: 0.001
0.487 =:2 0.022
0.355 + 0.008
0.254 + 0.007
Confideuce
0.409 + 0,006
0.324 = 0.006
0.227 = 0.007
0.526 = 0.008
0.416 = 0.012
0,295 + 0.007
SVHN
ISA
0.391 + 0.003
0.310
0.018
0.217 + 0.003
0.554 + 0.011
0.365 + 0.016
0.256 0.010
ISA-V2
0.455
0,006
0.357 0.010
0,243 = 0.003
0.516 = 0.007
0.130 + 0.014
0.307 0.027
Results and discussion. In Table ! we present the SR for Confidence (ERM) and the proposed ISA.
Following the setting in section 7.1 we inject different levels of uniform label noise into uninformative data and informative data separately, denoted by Ti and TU, corresponding to label noise ratio for informative and uninformative data respectively. Parameter À in Definition can be calculated accordingly: x 0.9-T1
1.8
1.8'
There are two main empirical observations from Table | First, ISA consistently maintains smaller SR than Confidence (ERM) across different scenarios. This result is consistent with Theorem 2 suggesting that upweighting informative data leads to improved risk on informative data. Second, we observe that both methods exhibit performance deterioration as the noise gap becomes smaller, which is suggested by the sample complexity in our thcorem. Our thcorem suggests that the sample complexity increases as the label noise ratio gap À decreases. When ā vanishes, informative date becomes less distinguishable from the uninformative data, and thus learning g becomes more challenging.
7.3 Experiments Using Real-world Data for Qs
Dataset. Tn this section, we report our empirical study on 3 publicly-available datasets: (1) Oxford realized volatility (Volatility) dataset (Heber et al.] (2009), (2) breast ultrasound images (BUS) (Al-Dhabyani et al., 2020), and (3) lending club dataset (LC) (Lending Club. 2007). The experiments aim to demonstrate the palential of the proposed algorithm in real-world application and its advantages in selecting useful information out of noisy dataset. The data description and respective train-test splits are presented in Table I1 in
section D.1
Experiment Details. We use the same network architectures, software and hardward as we do in section 7.1 7.2 For Volatility and LC,, we use the same Adam optimizer with learning rate (1c-3), weight decay rate (le-4) and batch size 256. For BUS, due to its limited sample size, we use smaller batch size (16) and reduce learning rate (1e-4) accordingly. For all three datasets, we train each algorithm for 50 epochs and reduce the learning rate by half at epochs 15 and 35. Each experiment is repeated 6 times with random seeds 77, 78, 79, 80, 81 and 82.
Specifically, for DeepGambler the default pappe--er o = 2.2 gives unreasonably poor performance on LC and BUS. We find that sctting o 1.5 gives the best performance and hence we use this value for DeepGambler in these two datasets. For other baselines as well as our method, we use consistent hyper-parameters across all experiments. They are listed in Table in section D.I.
12

---

Evaluation Metric. Unlike syuthetic experiments, in the real-world datasets, the ground-truth binary labels that distinguish whether data is precision/recall are not applicable. Instead, we report the selective risk of each algorithm given different coverage levels. Specifically, we pick testing data points that have top% coverage selective confidence given by each selector, and calculate the testing selective risk at different coverage levels accordingly.
are not available. Hence metries like
Results and Discussion. From Table we can see that the proposed method gains competitive performance against other baselines at low coverage levels. This suggests that our method is especially good at picking out strongly informative data. Such low risk regime can be captured by our selector loss, leading to lower selective risk, which is consistent with the conclusion in Theorem
Table 2: Woal-World Experiments: Selective Risk v.s Coverage
Dataset
Coverage
Coufideuce
SLNet
DeepGambler
Adaptive
Opeside
ISA
ISA-V2
0.001
0.000
0.000
0.200
0.000
0.267
0.121
0,000 I 0,000
0,050
0.055
0.000 1 0.000
0.000 1. 0.000
0.100
0.097=
0.008
0.122 + 0.003
0.328 = 0.015
0.168 : 0.008
0,113 = 0.006
0.091 + 0.008
0.083 = 0.001
Volatility
0.200
0.133 = 0.004
0.198 1 0.002
0.326
0.009
0.225 i 0.012
0.159 : 0.003
0.136 1 0.002
0.127 : 0.008
0.500
0.218
0.004
0.316 0.003
0.327 0.005
0.290 0.004
0,239 0.003
0.227 + 0.002
0.225 = 0.001
0.900
0.309 0.001
0.332 + 0.003
0.327
0.001
0.321 + 0.001
0.314 + 0.001
0.312 + 0.002
0.310 + 0.001
0.999
0.327=
0.003
0.333 + 0.003
0.327 = 0.001
0.329 + 0.001
0.332 = 0.002
0.328 + 0.001
0.324 + 0.002
0.001
0.026
0.032
0.167 + 0.408
0.000
0.000
0.167 + 0.408
0.052 0.036
0.000 + 0.000
0.000 = 0.000
0.100
0.026
0.032
0.396 0.156
0.156
0.147
0.083
0.078
0.052 + 0.036
0.020 + 0.031
0.010 + 0.026
BUS
0.200
0.026
0.032
0.422 0.134
0.151
0.057
0.089
0.094
0.052 + 0.036
0.021 + 0.026
0.005 + 0.013
0.500
0.038
0.029
0.432
0.062
0,184
0.046
0.135
0.140
0.052 0.036
0.058 1 0.019
0.017 1 0.016
0.900
0.085
0.019
0.303 + 0.039
03133 + 0.035
0.200 + 0.109
0.093 + 0.023
0.098 + 0.018
0.093 + 0.022
0.999
0.109
0.016
0.382 0.040
0.121
0.032
0.114 1 0.026
0,219 : 0.107
0.109 1 0.013
0.118 + 0.024
0.001
0.829
0.052
0.410 1: 0.142
0.362 1 0.058
0.357 1 0.072
0,106 1 0,036
0.061 1 0.012
0.219 1 0.1668
0.100
0.201
0.005
0.370 + 0.102
0.419 0.031
0.281
0.023
0,207 = 0.036
0.151 + 0.007
0.151 = 0.021
LC
(1200)
0.008
0.393 0.077
0,419 0.023
0.014
0,228 = 0.038
0,214 1
0.212 =
0.500
0.312
0.006
0.421 + 0.030
0,114 0.011
0,241 + 0.007
0,305 0.030
0.333 + 0.005
0.362 = 0.011
0.900
0.385
0.001
0.399 + 0.009
0.118 = 0.001
0.229 + 0.001
0.373 = 0.021
0.389 + 0.006
0.127 = 0.009
0.909
0.398
0.004
0.309 0.010
0.421 0.004
0.231 0.003
0.019 0.386
0.305 + 0.007
0.427 + 0.008
7.4 Ablation Study
In this section we present ablation studies about 1) the sensitivity of the algorithm's performance to the choice of
and 2) aforementioned practical implementations. We first present the results MNIST given different hyper-parameter combinations. In section 7.1 we set , = 3.0, AT and pre-train epochs to be 10. In this section, vary each of them one-by-one while fix the rest of them. The results are presented in Figure i in Appendix. We can see that the proposed algorithm's 's performance is robust against the choice of syper-parameters We next present an ablation study on the aforementioned
practical implementations in Table g Recall that we have the following relaxations: (1) moving average estimation of sample weight (MAW); (2) moving average of soft-label for informative data; (3) classifier and selector sharing the same backbone and the use of focal loss instead of cross-entropy. We systematically incorporate each module one at time, evaluating their marginal contributions to the performance on SVHN in the high label noise setting (T = 0.3 and TU 0.6). Results shown in Table 3 For Algorithm D the best run can get AP = 0.87 but the vanilla algorithm is not stable and end up getting large variance in its performance. In comparison, the relaxed version get much more stable performance. The ablation study suggests that cach relaxation strategy helps stabilizing the algorithm and make it robust against variance from stochastic approximation.
Table 3: Ablation Study on Implementation Relaxation. Method
AP
SR
Amgorithm
0.778 + 0.131
0.576 + 0.034
+ MA-Weight
0.763 + 0.094
0.573 + 0.019
+ MA-Soft (ISA)
0.850 + 0.020
0.554 + 0.011
Focal Loss (ISA-V2)
0.906 = 0.000
0.516 + 0.007
13

---

8 Conclusion and Future Work
In this work, we take the first step towards principled learning in domains where a lot of data is naturally uninformative/higllt noisy and should be discarded in both learning and inference stage. We propose a general noisy generative process that formally describes such setting. Supported by theoretical guarantees, a novel loss is designed for the training of the selector model. Based on this loss, we design a practical algorithm that jointly learns the predictor and selector. Empirical analysis demonstrates the effectiveness of our method. There are several directions for future work:
In the current framework, the fundamental principle of proposed method distinguishing the uninformative data points and to discard them during the training process. While this approach may enhance the statistical efficiency and risk associated with informative data, it may under-utilize the polential of data from representation learning perspective- - for instance, while the uninformative labels are useless by definition, the discarded samples themselves may still be helpful for representation learning. This can be achieved through methods similar to those presented in studics like (Sohn et al. 2020 et al. 2021).
using methods similar to (Sohn et al. 2020 Li et al. 2021
The Noisy Generative Process can be potentially generalized to solve different problems, such active learning (Cohn et al. 1994) and out of distribution generalization (Arjovsky et al. 2019
Bridging the gap between Algorithm and Theorem presents main challenges. Firstly, analyzing the disparity when minimizing cross-entropy surrogate for binary classification. We are optimistic that the H-consistency bound framework introduced in (Awasthi et al. 2022 could be
utilized for this purpose. Secondly, examining the use of Stochastic Gradient Descent (SGD) instead of the Empirical Risk Minimization (ERM) oracle in Theorem 2 In this regard, we believe that tools for analyzing SGD in bi-level optimization problems
(Chen et al. 2021 could be applied.
We look forward to these extensions.
9
Acknowledgement
We thank anonymous reviewers for their constructive feedback, improving the quality of this work. This effort was partially supported by the Intelligence Advanced Research Projects Agency (LARPA) and Army Research Office (ARO) under Contract No. 2911NF Any opinions, findings, and conclusions in this paper are those of the authors only and do not necessarily reflect the views of our sponsors.
References
Durmus Alp Emre Acar, Aditya Gangrade, and Venkatesh Saligrama. Budget learning via bracketing. In International Conference on Artificial Intelligence and Statistics, pp. 4109-4119. PMLR, 2020.
Walid Al-Dhabyani, Mohammed Gomaa, Hussien Khaled, and Aly Fahmy. Dataset of breast ultrasound images. Data in brief, 28:104863, 2020.
Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprint arXiv:1907.0255 2019.
Marco Avellaneda and Jeong-Hyun Lec. Statistical arbitrage in the us equities market. Quantitative Finance, 10(7):761 782, 2010.
Pranjal Awasthi, Maria-Florina Balcan, Nika Haghtalab, and Ruth Urner. Efficient learning of linear separators under bounded noise. In Conference on Learning Theory, pp. 167-190. PMLR, 2015.
Pranjal Awasthi, Maria Florina Balcan, and Philip M Long. The power of localization for efficiently learning linear separators with noise. Journal of the ACM (JACM), 63(:):1 27, 2017.
14

---

Pranjal Awasthi, Anqi Mao, Mehryar Mohri, and Yutao Zhong. H-consistency bounds for surrogate loss minimizers. In International Conference on Machine Learning, pp. 1117-1174. PMLR, 2022.
Peter L Bartlett and Shahar Mendelson. Empirical minimization. Probobility theory and related fields, 135(3): 311-334, 2006.
Peter I. Bartlett and Marten H Wegkamp. Classification with a reject option using a hinge loss. JMLR, 9(8), 2008.
Peter L Bartlett, Olivier Bousquet, and Shahar Mendelson. Local rademacher complexities. The Annals of Statistics, 33(4):1497-1537, 2005.
Avrim Blum, John Hopcroft, and Ravindran Kannan. Foundations of data science. Vorabversion eines Lehrbuchs, 5, 2016.
Anselm Blumer, Andrzej Ehrenfeucht, David Haussler, and Manfred K Warmuth. Learnability and the vapnik-chervoennkis dimension. Journal of the ACM (JACM), 36(4):229-96, 1989.
Olivier Bousquet. Theory of classification: survey of recent advances. ESAIM Probability and Statistics, 2004.
Olivier Bousquet, Stéphane Boucheron, and Gábor Lugosi. Introduction statistical learning theory. In Summer school on machine learning, pp. 169 207. Springer, 2003.
Tom Bylander. Learning linear threshold functions in the presence classification noise. In Proceedings of the seventh annual conference on Computational learning theory, pp. 347, 1994.
Tianyi Chen, Yuejiao Sun, and Wotao Yin. Closing the gap: Tighter analysis of alternating stochastic gradient methods for bilevel problems. Advances in Neural Information Processing Systems, 34:25294 2-25299--53077, 2021.
Chi-Keung Chow. An optimum character recognition system using decision functions. IRE Transactions on Electronic Compulers, (4):247-25)) 1957.
CK Chow. On optimum recognition error and reject tradeoff. IEEE Transactions information theory, 16 (1):41-40) 16, 1970.
David Cohn, Les Atlas, and Richard Ladner. Improving generalization with active learning. Machine learning, 15(2):201 -211, 1994.
Corinna Cortes, Giulia DeSalvo, and Mehryar Mohri. Learning with rejection. In ICALT, pp. 67 07-82, 2016.
Jacob Devlin, Ming-Was Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. orXiv preprint arXiv:181.004555 2018
Iliss Diakonikolas, Themis Gouleakis, and Christos Tramos. Distributtoo--indeeedddd pac learning of halfspaces with massart noise. Advances in Neural Information Processing Systems, 32, 2019.
Ilias Diakonikolas, Vasilis Kontonis, Christos Tzamos, and Nikos Zarifis. Learning halfspaces with massart noise under structured distributions. In Conference Learning Theory, pp. 1486-14680 1513. PMLR, 2020.
Richard M Dudley. Uniform central limit theorems, volume 142. Cambridge university press, 2014.
Andrzej Ehrenfeucht, David Hanssler, Michael Kearns, and Leslie Valiant. general lower bound on the number of examples needed for learning. Information and Compulation, 62(3):247-26)) 1989.
Ran EI- Yaniv et al. On the foundations of noise-free selective classification. JMLR, 11(5), 2010.
Giorgio Fumera and Fabio Roli. Support vector machines with embedded reject option. In International
Workshop on Support Vector Machines, pp. 68 82, 2002.
Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In ICML, pp. 1050-1055 2016.
15

---

Aditya Gangrade, Anil Kag, Ashok Cutkosky, and Venkatesh Saligrama. Online selective classification with limited leedback. Advances in Neural Information. Processing Systems, 34, 2021a.
Adilya Gangrade, Anil Kag, and Venkatesh Saligrama. Selective classification via one-sided prediction. In International Conference on Artificial Intelligence and Statistics, pp. 2179 2187. PMLR, 2021b.
Adilya Gangrade, Anil Kag, and Venkatesh Saligrama. Selective classification via one-sided prediction. In International Conference on Artificial Intelligence and Statisties, pp-
PMT.R, 2021c.
Yonatan Geifman and Ran El-Yaniv. Selective classification for deep neural networks. In NewrIPS, 2017.
Yonatan Geifman and Ran El-Yaniv. Selectivenct: A doep neural network with an integrated rejoct option.
In ICML, pp. 2151-2159, 2019.
Yves Grandvalet, Alain Rakotomamonjy, Joseph Keshet, and Stéphane Canu. Support vector machines with a reject option. In NewrIPS, 2008.
Steve Hanneke. Theoretical foundations of active learning. Carnegie Mellon University, 2009.
Steve Hanncke and Liu Yang. Minimax analysis of active learning. J. Mach. Learn. Res., 16(1):3487-3602 2015.
David Haussler. Sphere packing numbers for subsets of the boolean n-cube with bounded vapnik-chervonenkis dimension. Journal of Combinatorial Theory, Series A, 69(2):217-222 1995.
Kaiming Hc, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual lcarning for image recognition. Proceedings of the IEEE conference on compuler vision and pallern recognilion, pp. 770 778, 2016a.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition.
CVPR, pp. 770-778, 2016b.
Heber, Gerd, Lunde Asger, Shephard Neil, and Kevin Sheppard. Oxford-man institute's realized library, 2009.
URL https: oxford-man, ox. uk/data
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4700-4708, 2017.
Lang Huang, Chao Zhang, and Hongyang Zhang. Self-adaptive training: beyond empirical risk minimization. Advances in neural information processing systems, 33:19365 19376, 2020.
Heinrich Jiang, Been Kim, Melody Y Guan, and Maya R Gupta. To trust or not to trust classifier. In NewrIPS, pp. 5546-5557, 2018.
Adam Tauman Kalai, Adam R Klivans, Yishay Mansour, and Rocco A Servedio. Agnostically learning halfspaces. SIAM Journal on Computing, 37(6))1777 1805, 2008.
Adam Tauman Kalai, Varun Kanade, and Yishay Mansour. Reliable agnostic learning. Journal of Computer and System Sciences, 1888:1448) 2012.
Michael Kearns and Ming Li. Learning in the presence of malicious errors. SIAM Journal on Computing, 22 (4):807 837, 1993.
Michael J Kearns, Robert E Schapire, and Linda M Sellie. Toward efficient agnostic learning. Machine Learning, 17(2-3):115-141, 1994.
Alex Kendall and Yarin Gal. What uncertaintics do we need bayesian deep learning for computer vision? In NewrIPS, pp. 5580 5590, 2017.
Beata Beigman Klebanov and Eyal Beigman. Some empirical evidence for annotation noise in a benchmarkod dataset. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Associalion for Computational Lânguisties, pp. 438 446, 2010.
16

---

Adam R Klivans, Philip M Long, and Rocco A Servedio. Learning halfspaces with malicious noise. Journal of Machine Learning Research, 10(12), 2009.
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.
Michel Ledoux and Michel Talagrand. Probability in Banach Spaces: isoperimetry and processes, volume 23. Springer Science Business Media, 1991.
Lending Club. 2007 through current lending club accepted and rejected loan data, 2007. Lending Club platform is retired at 2020. The data is retrieved from Kaggle, https://www. com/datasetst
hordsforttheuise//endddg
Junnan Li, Caiming Xiong, and Steven CH Hoi. Comatch: Semi-supervised learning with contrastive graph regularization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9475 9484, 2021.
Shaojie Li and Yong Liu. High probability generalization bounds with fast rates for minimax problems. In International Conference on Learning Represenlalions, 2021.
Tsung-Y Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. Tn Proceedings of the IEFE international conference on computer vision, pp. 2988, 2017.
Ziyin Liu, Zhikang Wang, Paul Pu Liang, Russ Salakhutdinov, Louis-Philippe Morency, and Masahito Ueda. Deep gamblers: Learning abstain with portfolio theory. In NeurIPS, 2019.
David Madras, Toni Pitassi, and Richard Zemel. Predict responsibly: improving fairness and accuracy by learning to defer. Advances in Newral Information Processing Systems, 31, 2018.
Pascal Massart and Élodie Nédélec. Risk bounds for statistical learning. The Annols of Statistics, 34(5): 2326-2366, 2006.
Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar. Foundations of machine learning. MIT press, 2018.
Hussein Mozannar and David Sontag. Consistent estimators for learning to defer to an expert. In International Conference on Machine Learning, pp. 7076-7087. PMLR, 2020.
Hongseok Namkoong and John C Duchi. Variance-based regularization with convex objectives. Advances in neural information processing systems, 30, 2017.
Feng Nan and Venkatesh Saligrama. Adaptive classification for prediction under budget. or Xiree preprint 1 10194, 2017.
Nagarajan Natarajan, Inderjit S Dhillon, Pradeep Ravikumar, and Ambuj Tewari. Learning with noisy labels. Tn NIPS, volume 26, pp. 1196-1204, 2013.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Ng. Reading digits natural images with unsupervised feature learning. 2011.
Chenri Ni, Nontawat Charoenphakdee, Junya Honda, and Masashi Sugiyama. On the calibration of multiclass classification with rejection. arXiv preprint Xit.:1901.1063 2019.
Bernt Øksendal. Stochastic differential equations. In Stochastic differential equations, pp. 65-84. Springer, 2003.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advunces in newral information processing systems, 32, 2019.
Tim Pearce, Felix Leibfried, and Alexandra Brintrup. Uncertainty in neural networks: Approximately
bayesian ensembling. In AISTATS, pp. 234-244- 244, 2020.
17

---

Anupama Reddy, Jenny Zhang, Nicholas S Davis, Andrea. B Moffitt, Cassandra L Love, Alexander Waldrop, Sirpa Leppa, Annika Pasanen, Teo Meriranta, Marja-Liisa hanalaiinnnn-nnnndeeeeee et al. Genetic and functional drivers of diffuse large b cell lymphoma. Cell, (71 494, 2017.
Takaya Saito and Mare Rehmsmeier. The precision-recall plot more informative than the roc plot when evaluating binary classifiers on imbalanced datascts. Plo8 one, 10(3):c0118432, 2015.
Norbert Sauer. On the density of families of sets. Journal of Combinatorial Theory, Series A, 13(1):145 147, 1972.
Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. Fixmatch: Simplifying semi-supervised learning with consisteney and confidence. Advances in neural information processing systems, 33:596 608, 2020.
Sunil Thulasidasan, Tanmoy Bhattacharya, Jeff Bilmes, Gopinath Chennupati, and Jamal Mohd-Yine Yusof. Combating label noise in deep learning using abstention. arXie preprint arXiv:1905. 2019.
Ruey S Tsay. Analysis of fouancial time series, volume 543. John wiley & sons, 2005.
Leslie G Valiant. A theory of the learnable. Communications of the ACM, 27(11):1134-1142 1984.
Tim Van Erven, Peter Grunwald, Nishant A Mehta, Mark Reid, Robert Williamson, et al. Fast rates in statistical and online learning. 2015.
Vladimir N Vapnik and A Ya Chervonenkis, On the uniform convergence of relative frequencies of events their probabilities. In Measures of complexity, pp. 11 30. Springer, 2015.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkorcit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Atlention is all you need. arXiv preprinl 1rXiv:1706.088 2017.
Mathukumalli Vidyasagar. Learning and generalisation: with applications to neural nelworks. Springer Science & Business Media, 2013.
Marten Wegkamp, Ming Yuan, et al. Support vector machines with reject option. Bernoulli, 17(4):
1368 1385, 2011.
Jon Wellner et al. Weak convergence and empirical processes: with applications to statistics. Springer Science & Business Media, 2013.
Yair Wiener and Ran Elniv.... Agnostic selective classification. NewrIPS, 24:1665 1673, 2011.
Songbai Yan and Chicheng Zhang. Revisiting perceptron: Efficient and label-optimal learning of halfspaces. pp. 1056-1066, 2017.
Bin Yu. Assouad, fano, and le cam. In Festschrift for Lucien Le Cam, pp. 423-444 435. Springer, 1997.
Chicheng Zhang and Yinan Li. Improved algorithms for efficient active learning halfspaces with massart and tsybakov noise. arXiv preprint Xiv:21102 05312, 2021.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. In ICLR, 2017.
Chong Zhang, Wenbo Wang, and Xingye Qiao. On reject and refine options in multicategory classification. JASA, 113(522):730-7455 2018.
A Theoretical Results Details
In this appendix section, present the missing proofs well additional theory results.
18

---

A.1 Preliminaries
We describe the risk for selector loss on (æ. X x CR x (+1,-1}.
( (11(((()) - =}] R(g:f.8) == Ez.y [ +))
(15)
The choice of 3 should ensure that given Bayes optimal classifier (*),, (') is the minimizer for the selector risk R(g: f*, B). We have that given f*, the risk gap between any selector g and g" is Ríg; f) : ,) could be written as:
= Ea.y
[1(0(x) = +
11)((())) # +y}
+1(g(z) * = +10001/0111111110000000000111110
=n
- Em..g
チ4} (16)
|0000 + = m
+
((x)
3 +
((x)
E
3
+
1(g(z)=
2
4
2
EE
[3
X(x)
8
1
Xxx)
j (g((z)) + 1)1(00))))=)00111))))0)))))))))) =1 1)
2
4
2
Since X(x) data dependent, to ensure that R(g. f B) 2 Rig": f", ,)) for all g E 9, it suffices to pick + Amm)
( which implies that it suffices to pick 8 B
- - Ma) 2 0 and
- +
M=)
-3
Mmm)
2 2 0. So we need _ 2 sup 3-2M() and 3 < inf 3+2M[= 1-2x(=)
2
1 1+2x(x)
3 23 3+25
1122 1-2X
Assuming X(x) 2 X, we pick 33 within certain margin of the above interval: by picking 3-22 + +X, 3+28
1+2A
1-22
1-4X-
we have
R(g"t 2
4(1 2X) (x(l(g(<) + g (x))
(17)
Note that if = 1. 2. the above interval for B is co).
A.2 Proof of Information Theoretic Lower Bound
In this section quantify the hardness of recovering g", from an information theoretic perspective. We define a generative process that conforms with the noisy generative process defined in I and show sample lower bound for finding a selector given samples generated from this process.
Let X: ele E (el (e.,
e". e²), |r| S 1} where el represents the j-th cannonical basis. So X' consists of r-scaled basis vectors. Let y= = [+1,-1) samples are drawn from X' x y Rd [+1,-1}. Let be vector of ones, 1. For our lower bound construction we define f" follows for E X.
fx) 21 x > 0} 1 1.
So |g] = 2d and G contains (x). Let a (a oo).....o))) E (+1.-11) be a d-dimensional Rademacher vector, a (0,1). We define 9h E G follows
Let G be the hypothesis class that contains all functions g: x [0,
g3 (x)
Σ 1 1 ed +0 2 11 (1- //2 sa} -a)-(1++)))))) )')2)) 1=1
19

---

To clarify the definitions above, suppose for each j 1,...,d,, d, el is the vector with the jth entry being one and the other entries zero. Let S { e²} be all the vectors in X with non-zero jth. Then for æ D we have that f*()) 1 if T positive and -1 if is negative. Moreover, g²(x) L- if |||| 1211-a it is 1to' if [|]l a, and it is otherwise. In other words, if o² -1, the informative part of S is [ 2 1 - ar otherwise the informative part of ! becomes {~|||=<|| S a). Moreover, considering Definition the domain Da is uy=1 Let San {(x,, u)]i= be generated from following process which we denote as Q:
Unif(+
g6 = = 1( Te + #0]1 2 1
at a). (1 0/)/2
/=1
1[|=0] sa) (1+ o²) /
Generate S = ( 14) = according to:
j 1- j Unif(2,..., d3
with prob 1 - with prob f
(18)
Unifl-1,11
e
with prob 3 + (2g (1) (x) with prob + (2g 1
fx),,
Process Q follows process] Let A be any (potentially randomized) algorithm that takes dataset So as input where So,n is generated from the process described in Equation 18 Let g be the hypothesis ouput of algorithm A, i.c. 3(-) = A(S For a parameter we define the risk on algorithm A
R(A(S..) = R(ā(x), ,*, 8) = E., (11))) u)) 117()) + 1)1(f( = -3].
(19)
Theorem 3. Let B and be any real numbers where B E 1 - T-4X= and 0 t<e0<
 X. For any algorithm A that takes samples Sn from the noisy generative process as in Assumption foor some integer n and outpuls a selector R(S,) we have the following: If the risk gop is bounded by 5(142x]' i.e.
3+22
Es, (R(A(S))))) R(g", f00 B)] 8(1 + 2
then A requires
(og(()) Ae
many samples, i.e. n log( Gl A=
Proof. The lower bound construction is the process defined in Equation 18 Let ğ = A(Sx) be the output of A. From equation 17] the risk gap R(g. f. R(g", f*, ()) averaged over or and So, can be written follows.
Eg Es.. [R(G, (( - RRga (*,)))
B Es....
Em [1 (7()) + oe (z))
4(1 + +2))
Ā 4(1 + 2X)
d
FS Eso..
Σ
P i E P [px) + (6( E 2i 0
(20)
1=2
Σ
E Eso..
Pr [](x)) + ( E 22] [
4(1 + 2d)d
1=2
20

---

In the last inequality the fact that Pa|æ 2 c/(A 1)) 2 c/(Ad) for 2.
Let o0 be a Rademacher vector conditional on coordinates {l,.... - 1,j + +....d Let ol-3) be vector equal to except at the jth entry. We drop the n subscript from Sa, for simplicity. Above equation becomes:
Es, [R(O. f) - R(ga. f"" ,5)
E
Es.
Pa Pm(((x) # (x) E E
8(1 2X)d
j=2
x E 22' old
Es |(-)
Pr [
# go (x)
(21)
Σ
Ps m [[x) * to (ar) EN
2d-1
//E(+1,-1]==- 8(1 2A)d
1=
+ JPs. (-5) a [xx) + (0) E 22]
We make our notation more specific. Let A(Sa) = Do and A(Sa(-5)) = Do-s. For simplicity, by -o-, we mean (-1-))' Note that for all D2 9- (x) (x) could happen only when 2 ||>||| or | > So equation 21 becomes
Es R[G 1))
R(go, ,*,
1
Σ
So 20 (2r) + (z)) xc 2*
24-1
8(1 + 2
+ +Ps 8 (-1) m No- (xx) # IT ( x E mi]
d
as
Psy (x) (0)0) E | |||||| 2 - a)
2d-1
a a/iE(+1,-1)4 11 8(1 + 2X)d 1=2
(22)
+ s (-5) (x- + g0 (
22,
20 al
|,||,rr||||||
1
d
as
m P
Ps
E or, 21
2d-1
a // 1]d-1 8(1 2X)d
+ P 5 (-1) # D0- (x:) # - E S2
|
|||||||| 2
21

---

Next make Equation 22
independent of z.
Es [R ,), - R .3]
1
Σ
a0
P So [x) # (x))) E EN a 2 |||| or, ||||| 2 a]
2d-1
alvE|+1, 111-1 8(1 + 2
j=2
Do (x) * (x) x E 1 a | or, |||||| 21 - a}}
+ Ps (-1)) a
1
Σ
Es s m
E ss 2 or, ||=||| 2 --a
2d-1
0/E((11-1)á= 8(1 + 2
+ BBs
[1 (x) + -o (x)) x E 22 ||||| |
1
Σ
RS. 1 [āo # g x E E a 2 ||,|||, |||||| 2
(23)
2d-1
8(1 + 2
01E((11-1)á=1)
+ +Eg l- {] x
[1 * ge x 22
| or, ||||| 2 a1
d
as
Ps. [o + # + +Ps
[ + 9
2d1
11(+1,-1jd- 8(1 2X)d j=2
d
1
ac
1 - Pss. [3o # - + +Ps (((s-s) (Bo-s # 9
2d-1
dE(+1,-1]d-] 8(1 + 2
j=2
1
Σ
as
1 - (1Q(") - -
2d-1
8(1 + 2
oftE(+1, 11
/=2
where "!") and Q are the product distribution of n samples for Sy andS,-, respectively. The last step of incquality follows from the Le Cam's method. In the Equation we use the fact that for all s.t., || S a or ||=||| 2 1 a, [ (a:) + (x)) 1s * gh } is free of
Let Qo be the distribution of S, and Qu(-N be distribution of S.-1. The total variation distance can be bounded using the Hellinger distance, which is denoted as M(.,)). Below we bound the TV distance using Hellinger distance.
[[(!") -
"H(Q!") (
4
(24)
H²(Q8")
(n) Q.2) n
SVTHlQ0.
22

---

Now we bound the Hellinger distance.
J. (VQo (Qo1-1 2 dxdy
Jeenv Jeem.lelsa J (=) (VQo (z, (Qoi-n (xx, v)) dxdy
Isa
men 11 121 121 / (VQo(z,y) v (orl-1) dxdy
(25)
(vQr(z,)) (001-1)(2,,)) dady
/EE/ 1/ ISa (x)
2 dxdy (vQo(x,)) v -Qoi-11 (x, w)
+
mEAI || 12 (x)
3
1
3
A
1
1
1
2
+
+
V
2
4
2
4
4
3aet
d
Thus we can bound the total variation distance as:
|1 2'7 iy llv S 3
(26)
Note that from inoquality 23 incquality
24 and incquality 26 we have
E Es, [R(A(Sa),9) R(g". 8 B)
= =E. Es, [R(]. f", ..) RRlg .
ff" 31
(27)
1 00
(1-1 V 3ras
(1-V
2
d 8(1 + 2A)
d
Above implies
sup o1{
Rf'"" f8, 2 Es [R(A(S.), 8) - -R(g", f B)1
d
00
1
3naed
d 8(1 + 2X)
Since | = 2d, any algorithm A needs least !
log [0) Asa
number of samples so that there is hope
achieve
QE
sup
- f" S
32(1 + 2A)
Replacing with finishes the proof.
Remark 4. From the second inequality in Equation 20 it can be observed that the construction of information theoretic lower bound for risk function R)g,3) can also be applied to the construction of an ([oog(9)) (re))) sample complerity lower bound for [x(g(() + g" (æ)]. Thus our Corollory ] also achieves minimar rate for recovering g" for family of Noise Generative Process.
Sinnimar-optmall optimal
23

---

A.3 Proof of Sample Complexity Upper Bound
Here we prove Theorem iin which we bound the risk gap R(g; ,3) R(g": 3)) Recall that the empirical version of the selector loss is
Σ 1 71
+ wi = 1]1[[(a:)) =vi
Rs. (g:
Our high level approach is as follows. We first analyze the gap between Rs, (0: () and Rs, (g"; 3) and provide an upper bound for it. Then we use this upper bound to get an upper bound for the gap between (",8) and R(g";J"; f',,8 B) using concentration properties and Bernstein inequality.
CASE I: J(0 E F and (af(() + Mx)a] 83 with probability at least 1 6. To upper bound Rs, (0: 8) Rs, f (), use Rs, (0: f. B) and Rs, f. middle step. Since is the empirical risk minimizer, we have
Rs. G7 f. S Rs, (0"; F.3).
(28)
Next we leverage on the fact that f is consistent with f to establish an inequality in following fashion:
Rs, (o: Rs, (g: B) + const
Note we have that
Rs, (T: F,3)
Mai) + Maxs)
(1 1]1(((a;)) = -wi)
7
(29)
= =vil
Recall that
Rs. (3) n
( A 10(x)) - 1), -
So
Rs., (g: F.g)
Rs. (ā: f :3)
1 f(as)) + (ax)) (100))
+1 - = =vil
t=1
(30)
Σ f(xx)) + ((x)) (1(0(a)) = 1)11f(2)) + +1] = 1)1]((())))))) = =vil)
B 8-1 8-1 Σ
"
(... # M(.))
2 Rs, ( ("))
n
Recall that in the theorem assumptions we have Ex (x) + [*(x)) S & with probability at least - ō. By
83
if n 2 248² (og())1/5) have with probability al least 1-6 6, 1 Σ (((())) # f (w:)) f.
Lemma. 2
have
Rs. (o: f.3) 2 Rs. (3: f... -
(31)
With a similar approach we get that.
Rs. ( < Rss (0":
24

---

Thus using 28 the following inequality holds with probability at least 1 1-6 8
Rs. f S Rs. (g": ff ,, +
(32)
To get a bound for R( B) ff we first define ((g: f. x. x) fg(( = 1)1 + ot +
1 [g(x=) = (z) = y). Note that at this point we think of as fixed and so we have not included it in the arguments of ((.) for simplicity.
Observe that Rs. (0. 3) Σ. ((:: f", a2 v) and have the following simple inequality directly taken from 332
Ex. [e(g: f", x. v)] for any g. First we
R f" 3) ""
= (0i f f=f"==1 x. v) - Ez. y ff x.
(33)
<Rs. (f: - Rs. (0: f. 3) - (En "gg" f" x, - Eo f" x. v)) + E/2
By defining .9 = {(g;; 5 x, {g: f v) for any 9, we can express the above inequality as follows:
E / (ā: f x. v) Em, f w.
(34)
(g G. u. y) - En,y Ag : x. + 8/2
To bound 1 ΣE n
Allyg (i) Eu, Allg" ; v. with high probability over all Sn, we need to find a Al(g' . v) Ez [g ",,""g",0 x, y) that is true for all simultaneously with high probability.
bound on We have :
1=
n
PsN
3g E
Allg' g: 0i vi) - F AAlg" gi xz y)
2V AVar(A(g) g: v)) log (ggg(((//)))) /100)
}
2 8
log(191/5) +
n
(35)
Σ Psa.
Ag .i (i,. - E A g3
gEg
2V 2Var((a (((" gr v)) log( G)/0)
z (gg(())/5) +
n
Now to bound 1 E Allg", 9. F. y) En.y Al(g" g, x.
we use Bernstein inequality. For that we need to bound Vara. Allg" 9. x,,,))] y)]. First we expand Aelg" "l(g",g,20 x0 v.).
Allg". "o00.gg,00 v, -('" -(g;
=01(g"(x) -1000011--1
-=1()(x)= -100000-0111000000000--0-1111111
So we have
A All(g" 9. g,
= B² = =
(36)
())))(())+)))(0100))))) - - + (z) =1]]
x (0) # }g(z)}
25

---

Hence conclude that Var=.y|Ae(q)) 9., y)] E A(g" g9 & 5'EelLig (x) g(x)]. On the
other hand, Equation 17 implies that R(g: ,) B) - ;ig"; B) 2 1+27
- Te (æ) * g(x)]. Thus we can use
the following incquality achieve fast rate of convergence using the Bernstein Incquality:
V Var Varn.ylAllg" " g, < B + 2
(R(g: ,,, -
(37)
We use following version of the Bernstein Inequality, with XI, Xn iid random variable uniformly bounded by b:
b log(1/0) + 2V ar log(1/0)
P
X; E[X]
1 -
Using union bounds, the Bernstein Inequality implies that with probability for all g 9 simultaneously:
Ps..
Al :9;0",;;;,0 vs) Bn, 9: m vi]
i=1
(38)
3 log (191/0) +
2V arr (ar(A , v. v)) log (9 (91/0)
1
Thus by applying inequality 38 with 9 have:
Ps..
, (. - EnglDe(g" 9i x wl]
8
2Var(A(g T: n. v)) log (99/88)
log (1g1/5) +
21-8
By inequality
32
we have 1 Σ
Allg", x.) RR, ( ff, B) Rs. f*.)) 2 -E/2 holds with
probability 1 J. Note R(g: f), R(g": ; -Ez,p |E,gg . x. 3)]. So we have
Ps.
[R(:: f) - "('")
3 <= log (1g1/0) + 2,3 + 2A) f. f*, B)) R(g:: log (9//6)
e/2
An
21 - 28
The choice of 1 log ) ensures that with probability at least 1 25, R(G: (" R(g"; - S
Ae
CASE II: f(-) E F that satisfics Eslf(x) + (x)) E 22 s 83a with probability at least 1 - 8.
Note f only approximates (*) the informative support Ωt f(.) E F that satisfies [x[f(x)) + (x) E
21] < 3pa with probability at least 1 --6 8. For simplicity of analysis, we introduce a 'pseudo' version of (-) denoted as f. Let F be following hypothesis class:
{
[ f1((), a E Zr
, EFEFEEF
[2(()) a e a
and we let f²()) be:
['(()) = [(x)) a E Qr
(1)),, Ωt
26

---

By the choice of Clearly, f E F. Note such hypothesis class is only introduced in analysis and is potentially impractical. The cardinality of hypothesis class [F] I | The construction of f is to make Rig, 18) R(g", ""' ,3, 2 (,,) R(g", , ,), .) for all g g. To see this, we use the fact that f is the Bayes optimal classifier, which implies that Va, (gg" ()) whl (x(1(0((x)) & 1} + yHl. Subsequently, we
have Va, E, (a) + 11]1f((()) - wtl s (x()"((()) + (1)){f(x)) = wH|. So we have the following.
=:hp::/|hhhhhhloooooorrooooo---3 -
Bnm 14,,0006110100000000004010013330 (39)
+1(g(x)
3 X(x)
Ex
= -+)
2
4
2
[[3
X(x)
(1 {(a)
(10(0) # (1) - 13
+B
[1
f" R(g": f
Meanwhile (+) also satisfies the property that Exl/((x) + f(x)] S 50 with probability at least 1 1- 8. Thus
by Lemma 2 n 2 24 logí [ we have with probability at least - 6, 1/2 Σi= 11 (f(m,) # (xx)) S sS.
The rest of the proof is the same as the proof in CASE I by replacing f with f*, leveraging on the fact that f is to make R(g(z), f 3)
fr 2 R(g(z), B) - Rig (x),
A.4 Proof for Corollary G
It can be easily verified that B = 3 is in the interval
8
3-22 1+2X
A, min
3+21 A
1-4 10)].
1-2X
3, from 17 we have
f",
E [1(()) # g*(x))}
4(1 + 2A)
together with the conclusion in Theorem I that
R Se
we can conclude that Equation holds.
A.5 Theorem on improving selective risk
Theorem 4 (Improving Predictor Selective Risk ). Let S, = {(x,, y: ) be i.dd samples from the Data
Generative Process described in Definition under Assumption ] with f*(() E and g") G, [F < xc, 19 <
00. For any g(.) E 9 s.t., [1 (w) + g(x)H < e, let f arg min 1 En (()()) > 0} + w: JEF
Then for any e > 0, 8 > 0 such that the following holds: For n 2 max! 20 101
12 logg L
we have
Ex(1(((x) #
[xx) <
(40)
a
27

---

Proof. Recall that f= arg min Σ 1((() > 0}1(f(x;) * un with 3(-) G s.t., Ex(1(g" + [()))|
JEF
s, From the minimization property we have
H0( uu) n
+ v1
(41)
Note we have for any f:
1
ΣE (11((1))) 4md
(a) > 0}1(/(a)) f
1m1
((())) > 03 - (m)) > 0) [{f(x)) }4}
Σ
((((3)) > t0 (gg((a)) 0}
(0((,)) # (xx,)
(42)
For n log( 101
by Lemma 2 we have - ((ā(x;) # (z:)) S Ex[Lig (x) + [(*)M s 2e
with probability at least 8. Thus by inequality and inequality 41
we have both
(.)))) 0)1 * wi 22 S 7
(Tx.) (3: + m)
7
and
Σ (0(x)) 1) xv) - 1
# 9i + 2e.
with probability least --6 5. Summing up the above inequalities and using inequality 41 probability at least - 6:
we have with
(m:) 4 v1 S (z))))))))(( v
(43)
Next we bound for Eg, (g) > 0)(1((() + v) - 1(f + y
we first define A f, V1)
- + y11) for any f E F. Due to inequality
43
we have:
[x > 0H1(((0) - 10 ッHH
> (g(
f, 0, Ex (1 03A(/ F, 4s
(44)
To bound 1 ΣE (xg) > f) v.) E., (g) > ", with high probability
over all Sn, we use the Bernstein inequality to achieve the fast generalization rate (similar to the proof in Theorem
By the definition of A(f f. x. we have
- -
- 10)())) 0|\(<>)))))))))))))))))))))))))) # [())))
(45)
28

---

Let w(x) (z) [ _ -0) We obtain the following:
(xy(-(g((x) > Ba
(46)
4 (z) [0
Thus using equation
45 and inequality
16 we have
"arg. (a) 00 f:x, v)l & 4E 0) - + y|ll
(47)
Using an approach similar to the proof of Theorem one can show that n 2 16 - following inequality holds with probability al least 26.
suffices to guarantee
E [1 ( # (x)) se
(48)
which implies that
> of} S e
Eg
(49)
a
A.6
Proof for Theorem 2
Define wxx) (z) y/ 3 (2)((2g(()) Since X(x) 1 h, one can show P() + y/æ) 2
satisfies Massart condition with margin 5. Since n 2 l4
[i the conclusion that [x( + ['(x)) <
follows from 5.2-section in [Bousquet] 2004). The risk bounds on ÿ and f follows from Theorem and Theorem 4
A.7 Proof for controlling conditional risk [x( + (x)) E Sr]
Definition 4 (Growth Function Vapnik & Chervonenkis 2015)). Let G be the hypothesis class of function f and Fe....... ((f(()))... f(xn)) T (+,, The growth function is defined to be the marimum number of ways in which points can be classified by the function class: B sup 21 2.. |Fay.......!
Lemma 1 (Sauer Shelah Lemma(See (Blum et al.) 2016 Mohri et al. 2018 Sauer 1972 )). Let dgo be
the VC-dimension of hypothesis class G and let Bg (n) be the growth function, for all N,
duc
doe(g)
Bg(n)
11
en dgo
Theorem 5. For every e 0, O, under Assumption drawn d. from the Noisy Generative Process and
1 given set of samples Sn ((11))1), (Xn, (xn.yn)}
f = arg min JEF
Σ ((f(x) + v.),
if n is chosen such that
32 ( log( +lgg++ogg) log(!)
n >
2a²
then with probability at least 1 25 we have:
1 (1 - a) + 2ea. En.y {y] s
Furthermore the following holds:
Pa(s"(x) # f( s2 2e
29

---

Proof. We first bound the probability of the event that Ex,)((()) # vl s 221 a) 2ca
By Lemma
3 we have
E. [1)((=) + wHll 2 | S 43x m1 1 (f())) + v.)
Ps. sup JEF n
(50)
32(1o (2n)) we have 4By(2n)e # S á and so using inequality we
By selting / = are and n 2 have with probability of at least 1- ò:
Σ 1f(xa) vi Ex)(((f(xx) # wlll
n
dve(F)
The term By(2n) could be bounded by Lemma as B2 S
2nn dve(J)
Next we apply the fact that
== arg mig 1 Ei= L[f(x;) + v1.. We have:
Ex Ex [1)[1((()))+ w S ae + 1 Σ (((a)))) + f/ S ae + I(()(()) 4w}
(51)
By Lemma we have 1 Ei 1(f + w S 111 + ca with failure probability at most 8. Combining this with incquality 51 we have with probability at least 28:
(x)((((( + w/1 - (1 a) 2ear.
And this finishes the proof of the first claim. Next we prove the claim that:
of + f(x)) s 2e.
First note that PA-D, (xx) * [(x)) = PUND. (1( # (x) E 21). Using En.(11([aa) # mHl s 1/4 (1 - a) + 2Ea below we have:
[ [f(()) + y}l
=B (z()) + y)}
= -B [1f(x) y) x E Ayl P (aay) (x e pvl
1
(f(z) (f + v) l E Ω7) P (a.)) ( E 2t)
(52)
P (1
1 (1 a) + +ap [f(z) + ((()
< (1 a) + 2Ea
PPaND [1 (f(x) # (x))) x e 2,) s 2c
This finishes the proof of the second claim.
B Extention to VC-Class
order to leverage the margin condition of distribution to obtain a minimax-optimal generalization rate, we leverage on the Local Rademacher Average tool. Our analysis tool largely follows from (Bousquet et al., 20003 Bartlett al. 2005). Throughout this section, S and 2 represent as shorthand for the I and 2 that ignores universal constants.
30

---

Definition 5 (Lv-Coverin Number). Wellner et al. 2015) Let Iinn be set of points. A set of U S R" is E-cover w.r.t L2-norm of F Fim, if F, Eu U, s.t. V + [[]] f(x;)12 S E, where [u]; is the i-th coordinate of u. We define the covering number F,
Nz(E,F,21ma) == min(U]] U an e-cover of on Fim)
Let N2(e, be the mazimam cordinality of N2, F, Fin)) over all Zım. Formally N2, F. n) is defined as:
Ne(e,F,n): : sup min U is an E-cover of F on Fim BinEA"
Definition 6 (Local Rademacher Average (Bartlett et al., 2005 Bousquet et al. 2003)) Let di: be Rademacher sequence of length n, given samples {æ;, Vi the Empirical Local Rademacher Complexity at distributional and empirical rodius r 2 0 for the class F are defined as
1
RR Pf <r) Eoim
off(
sup
(E) n i=1
RnFF r) Eorn
SEF E. Ps 11 (x.))
sup
and their distributional Average as: R(F,Pf2 1) = Es, R (F, S r) and R(F,Paf r) Ess (F, f2 S r)]-
Definition 7 (Star Hull). Bartlell el al. 2005: is defined as
Bousquel el al. 2003)
The slar hull of set of functions F
+F {af: f E F, [0, 1]
Definition 8 (Sub-Root Function).
Bartlett et al. 2005 Massort & Nédélec, 2006: Bousquet al. 2003
A function vp R + IR is sub-root
is non-decreasing
is non-negative
wrr) fr is non-increasinnfor any
And say r" is fixed point of dh if 1b(r") r.
Theorem 6. [Risk Bound VC-Class] Lel Sn = (wx, ) be i.i.d samples from Dala Generative Process
described in Definition L under Assumption
1 with f()+) E and g"()) E g with VC- VC-dimension dyc(F) <
oo d9) < 20. Given X, let 3 E
-3 X, min( 3+21
I-2X - 10)]. For any f(.) E F, let 9 =
L142A
arg min (g: 3). Then for any 0, 8 0 such that the following holds: For gEg
2 mar (gv (og( + 8a log(t) (dvc(F) log( A + (og(()))
and for f that satisfies one of the following condition:
For any f(-) E F that salisfies E. Eml/(x) # (x)) & with probabilily at least 1 - 6,
For any F(-) E F that salisfies E. (m) + 1 (ær) E 2 pa with probability at least 8,
The following holds with probability at least 36:
R( f 3) =(g": 5)
Proof. The major difference from the proof for Theorem the fact that g and F are not finite hypothesis classes. To achieve fast generalization rate, we leverage the Local Rademacher Complexity Tool from (Bartlett)
et al. 2005
31

---

CASE I: F and Exff(z) + 5 2 with probability at least 1 1- 6.
We achieve equation 30 which is restated below exactly the same as in Theorem
Rs, (F:18)) Rs. (6: .)
+ + ((xi)
n
Then to achieve inequality we invoke Lemma x instead of Lemma 2 as F. a VC-class. particular, since 2 (idve(F)oo (o have that Σ 1[f(x,) # fz (glff()) # f*(x)) + Moreover,
from the case assumption we have Eg [f(x) + f (x)] S o with probability at least So with probability
at least d
Rs. ( f. 2 Rs. ( f" - e/4.
Similarly we get that with probability at least
Rs, (g": 3) s Rs, ()) +//4.
Thus following hold with probability at least 1 1- 8:
Rs. O0: 8) S Rs, ff: ()) 8
(53)
Next we turn to bound the risk gap using f R(g": f* using the concentration property of incquality 53. Similar to the proof in Theorem 1 for any f F, 9 define E(g: f. x. y) (l(g(x)
(z) 7 v + +1((() # (})({()) = v)- Based on E, we define following hypothesis class:
Aolog Al(g:g" x. = ((g: "=,,,,,,, x., v) Elg": x. (.,0)=9E.) }
(54)
To invoke Lemma 6 need to establish some hypothesis class H that satisfies condition Var [h] < [B]
Next show A o fo G satisfies the condition that Var[h] B[[s]] and thus can apply Lemma 6 with
H = A o lo G. To begin with, using the same approach as in Theorem we can obtain Equation 36 which is restated below.
leggg 9, v. S (g" (z) + {(x)}}
By Equation 36 we have that Var Algg", (llgg 9. σ. v)] < E. A l6g g. v) S (L + g(x))
On the other hand, Equation 16 implics that
R(g: 13, R(g"; ;)) 2
Eeg (g) * ((²))
1 2
Thus we have following holds:
(1 Var a.y (Alg:g"" ,x, < Ā
Eoz.y
(55)
Above variance bound allows invoking Lemma with H = A Aologg T(h) = [[A?] and B = (*(122)) satisfy the rest of the assumptions of Lemma we find a subroot function v(r) that
To
(r) > (1 + 27)
Ess. RR., (A((g:g"))) E H: E(h2] < <r]].
X
Where we shorten Al(g:g'. x. Al g²). To find (r)), we show some analysis on the Local Rademacher Average Es,, (R., E #: E1h r)] follows.
32

---

Es,, [Rn (Aooogrr) mn
sup
o ," a.)
geg,Ba. [ S
1=1
n
Es Es..
sup 8E9,2n (197 S1 72
og ";Al(gg'' ..,
(56)
1 ) >
sup g o, + (x:)))
Σ
7
(a) [()
Talagrand Contraction Inequality (Ledoux Talagrand
1991
In the last inequality we use Talagrand Contraction Inequality (Ledoux & Talagrand 1991 for which we need to show that (1) Al(gig") is a function of (g) # g"} and (2) Al(gigg)) is 3 Lipschitz in 1 ( # g*}. These conditions are satisfied as follows:
= =
g)A((ggg"))
and
Al(gngg") - A((g2:g))) (t)
(2..2.1))) 12) B # 92) (9) F9¹) - 199 # 9-)]-
This finishes the proof of Incquality 56 Now define the class g = 1 (g( (x), E g). The indicator function 1 (g(x)) + (x)) Boolean function taking g as input, thus dvc 9 S dg) Thus we have
(1 + 2X) ì
Es, (M((g(())))))) E H.: :H
(57)
30 + 2 X
Es. Rr (1(g(z) + (z)) [[(((()) # g*)())) s(]
Above implies that we can pick (r) to be
((r) 33 + 2
11 log
Es, RR [=1 [[[[g(x) + ("())] < <rl +
(58)
By Equation 50
Lemma 6 we have:
2 Σ =" + s n
1500Å
log log(1/6)(118 + ))
Ael g" x. v)
.22 r* +
(59)
n2
Where r" is the fixed point of (rr). By inequality
53 we have En Al(ā:g" :'0
24, U1) = -Rss (0:f",3)
Rs, (g"; f"33) () < e/2 holds with probability 1 - d. By Lemma we have r S 9o
dvc log
Plugging in
X²
Equation 59 have that 2 ( log tog(1))11)))) suffices to achieve E. [ 9" ar v) S Recall that Ex,,, Al (g: g" x, v)] R( .) R f) so this finishes the proof.
CASE II: f()) E F that satisfics E, [f(x)) + f E 22 S sag with probability at least 1 8.
=
The proof is similar to the one in Theorem except for that we need to bound the VC-dimension of pseudo hypothesis class F. Since f can be viewed Boolean function given [1()), f2(x=) input, with two hypothesis fi, f2 F, by Lemma 3.2.3 (Blumer al. 1989) we know (Fc(FF S (Fol The rest of the proof follows from the one in Theorem
Next we present our extension of information theoretic lower bound to VC-class. The lower bounds suggest that the risk bound in. Theorem light up to some logarithmic, factor.
33

---

restated below: For any f: Theorem 7. Consider the noisy generative process defined in Definition 1 with Up being G-realizable. Then for any S X, to achieve
Es. (R((())) - RRg", , 3315 8(1 + 2
with E
3-23 1+2A
X,
3+-23
any algorithm A will take at least dvc
many samples.
1-2X
I 1-4Xa]
(o)
Proof. The proof follows from the proof of Theorem B except for the fact that we need to have an upper bound on the VC-dimension of our hypothesis construction G. Since 9 consists of composition of interval hypothesis and each individual interval has VC-dimension at most 3. By Lemma 3.2.3 in (Blumer al,
1989 we know dgc I 6d log(()) which implies a log(dve(@1)Ar lower bound.
dy
Theorem (Improving Predictor Selective Risk for VC-Class). Let S (a V1) ())) be i.i.d samples from the Data Generative Process described in Definition | under Assumption [ with f*) F and g') E g, dvc(F) < oo, sv.dv(()) co. For any 9(-) E G 8.l., (x)) + [(x)) s, let. = arg min 1 Ei=1 1 (0(x)) > fer
0}1{f(z;) + vit. Then for any 0, 5 > 0 such that the following holds: if
2 max ( [Fc(FF) log( dv + log(3) dvc(g) log( dvc(o)) + (og()))
we have
Exlf(x) + ("(x)lg" + a
(60)
Proof. We use proof similar to the one in Theorem 4 up to Equation
42
(ā(x)) > (f;() # +w Σ (a,) (f(()( $ fv
1=1
x < 1 (0(x,) + g (x:)}
(61)
Since G is a VC-class, we will invoke Lemma instead of Lemma 2 Since n 2 dve log( dvelo)
H h) by
Lemma we have Dim1 (z))) A g" (xx)) Ex (g(()) + g (x) +e with probability at least 1-6. Since
Eg T + g" x H S e, by inequality 61 we have the following with probability at least 8.
a
(m))
# #v1
Σ (g)
0) + vi) 2
(62)
Similar to the proof of Theorem 6 next establish some hypothesis class H. that satisfics condition Varlh] < BB(h). We define:
A.
(63)
To invoke Lemma 6 need to establish some hypothesis class H that satisfics condition Var < BE]h).
Next show G A F satisfies this condition and thus can apply Lemma 6 with H G A A.F. Recall
Equation 47 in Theorem 4 shows that,
Var [4(g 03A(F ff v) < 4E
()(1((())) # fو 1z) *
where A(ʃi, f2, vi, = (z;) 4 - 1. So H satisfies the condition that Var(h] BE[
for B- = 4. To invoke Lemma 6 we let H. = A F. Thh E[A and B 4. Similar to the proof in
Theorem 0 we next find subroot function (r) that
(r) 2 4Es, [R,, (1 (g (mx) > (}A(3;')) H: Eph r]
34

---

To find (br), we show some analysis the Local Rademacher Average.
Es, (R Ao Frr] = mimm
sup
[=]g(())=0)(((+>>> vil
fEF |E7BB.,/110000))))))
Esmiiml
sup
(
Es Ti
sup (EFB))1) (x)
(a))))
[ag' Talagrand Contraction Inequality (edouxx Talagrand 1991
(64)
In the last inequality, we use the fact that (x;) > 0}A(S:1") 1-Lipschitz function of 1 (g)(x;) Of + f}. In particular, A) 1[ * fJA(f and
>
[1/g((;;) 0}((fi:g)) (()((()))))))))))))))))))))))))) > 0}A(f2:9")
(65)
<1(g"(z;) > 0)|A(fi:g") A(feig')
(m;) (m))>)][((/+>> ()]T(}i #) 1/2 y)
(1"(m)))00 #
==) f²) - (=)))))( 0}1 f')l
(66)
And so we can use Talagrand Contraction Inequality and this finishes the proof of inequality 64
Now define the class F. (z) 0}1{f()) # fz)), E F}. The indicator function [( (x)
>
(f) * (x) is a Boolean function taking f as imput, thus dvc 1 S dyc(F) (Vidyasagar 2013).
Thus we have
R.s RR, ()A(:))))) EEg (1(g"((())) 0)L[f(2) +
(67)
([) 0)1((()) f*())
Above implies that we can pick (r) to be
176 log n
(r)) [Rn : >=) # f(())] <r +
(68)
By Equation 80
have:
(gg((x)) > 0) v.) + 1500r + 176 176 log(1/8)
Ex. > =)
(69)
n
n
By inequality 62 have Σm Aff:f" :;
v; v1) 2s holds with probability 8. By Lemma we have
r" < dvc(J) logn Plugging in Equation 80 we have that n 2 (dve(T) log( dyo Hlgg(1/8) suffices to achieve
n Ex,y (zg" > oJA(F x. v)l S e. Similar to the proof in Theorem we have:
E
03l $ a
(70)
35

---

C Technical Lemmas
Lemma 2. Let S. = { wi)} be i.i.d sample from Data Generative Process described in Definition 1 For
every O, there exist ô > 0 such that if n 2 { the following inequality holds simultaneously for all f E F with |F| < oo with probability at least 1-
Σ (f(x.)) + (x,) +
# f((x)]++ +E
(71)
Proof. By taking union bound one can ensure that
Ps.
sup JEF
Σ
(()))) + (()))1]] ms]
1=1
n ((a)) ( # ( 2 ne)
SPs S
A FFEF:
P....
Σ
(z)) # (a)))) 2 no
JEF
We next apply the following version of Chernoff inequality with a 22 1: Let X = Ei X, where X, E (0, Then
a² E(X)) S exp ( a
P|X 2 + +x) S exp
-Tx((x))
++a a
P[x & (1 - a)xx) S exp (- E(X)) S exp --p
a²
"
2
So we have
P| - B(X) 2 aE(x)] S exp (x)
(72)
For any fixed f F, let X, = ((f(x;) # (x,)), and let a E/ [1 [f(x) + (x))) Then by inequality
72
we have
IP s,
[1(())) T. (f(x)) (x))
(73)
m.y (1 [1)(((((()) ( exp (-
<exp
Using 722 and setting |F| (Fleep(-nna /33 finishes the proof.
Lemma 3. Suppose S, ((x1, 11 (I Un)} are i.i.d sampled, and let T(f, x. E (+1,-1) be a
function. Let Ls. (f) n Σm= L(f. 34 y1) and L(f)= E,,g x, y)]. Given parameter such that
nt² > 26
then we have:
m12 P Ps. ND ssup Ls. (f) I(f) $ 4By(2n)e
fEF
Proof: For two sample sets S, and S,, if we have LLs, (f) L(f) 2 t and [Ls; (f) L(f) s 2 then we get that I.... Lss: 2 $. Let f be f that attains sup LLs., (f) - I(f)), one can verify that FEF
1 1(Ls. - 1(f) 2 1) 1 [[Ls ( I())= < '
(74)
< 1 Ls, (f) (f) 2 "
fEF
36

---

Taking expectation w.r.t Sn D and S D we have
Ls. (f) L(f)) 2 f] Ps; ~D (f) (f) = '
2 SI sup Ls. (f) -
PS sup FEF
feF
(75)
t 2
P. St D
sup Ls (f) Ls; (f)l
FEF
Next we lower bound | Ls. (f) L(f)) - { for any fixed f. Note that the choice of f is free of St since
Sy and Sn are iid samples. Since L(f E [-1,1]] and so Var(L(f, x, v)) s b²/ /4, using nt² 2 26 we have that:
t
4Var
Ps IIs, (f) L(DI
I 2
2
nl2
So we have L (f) I(π) S $1... A sup I.s. (f) I(Al
2 1/2
Let Fsa. S [+1,-11 -1)2n be
FEF
projection of F on Sn US,, combining this inequality with 75
we have
(s)(() - 1(f)20
FEF
<2P Sn 8/ //nD/suDsppp
Ls, (f) Ls; (f) 2 2
FEF
Ls. (ʃ) - Ls; (ss;(()] 2 2
=2 S f(a))Foan
sup
<2P S2n
Sm 8%
sup f(a))FF22
(f) - Lsy, (f) 2 |12nn] ]
<2P S2n
Ps. Sy [|. ( L,g (1)
t 2 1
(76)
1(x)EFB2
nt² [2[Fsan le
22P 82n
|52ml] ]] m12 [Fsm le
462
21P S2m
ssup San
in
[2
ur²
2ssup [Fsm. PS [e-
40
8mn
nf2
<2Br(2n)e
Lemma 4 (Hoeffding's Inequality). Let Zi, ... Zn be independent bounded random variables with Z, [a,b]] for all i, where -o <a co. Then for all t >> 0:
n Σ
2n12
P( n i=1
Z,
2
2 t) S 2c
4jz
(77)
Lemma 5. Consider a set of samples S {(x), yn) y1),... (z }n)} drawn i.d.d. from the Noisy Generolive Process and f in the hypothesis class FF satisfying f(x) E 1-1,+1) If:
3 3lg((t) aa²
Then have with probability at least 8 8-8
n n i=1
(0)))))) # m) S (1 a) + as
(78)
Proof:
37

---

The terms (ff( * ye) {0,1} for E [n] are set of independent random variables since (z,, V4) are independent. So we can use Hoeffding's inequality (Temma 4.. By setting b 1, = ore in Equation 77 the choice of ensures that
2nl a]
6log(0). Thus
Ps. N. |1
E n
1(f((a)) # Emyl + wtll 2
where we have
E B(ay))D, (x) fyll D( (x) + a E Pp Pyl (( (1 + u)fl 9r|Plp E Qt]
(79)
P(wEfty]: Since y labeled by coin flipping Dg
U: Since is labeled by f with Bayes Risk in f
2 (1 a)
This way have:
1 n Σ (0)))))) # vi 2/1 - -a] 2 ea) S Ss D. n
1
which implies that Equation 78 holds with probability at least 5-
Lemma (Theorem 3.3 in Bartlett et al., 2005 ). Let FF be a class of functions with range in [a, b)
and assume that there are some functional T : - R+ and some constant B such that for every E H. V V'ar(h) (h) S Th) S BE[[]. Let up a subroot function and r* be the fived point of th. Assume the salisfies, for any r 2 r"
((r)
BEs, Rr (h E H: T(h) Sr}
Then with Ca 704 and 02 = 26, for any K > 1 and every L >> 1 with probabilily at least - -
K
K r" B
t(11(b a) c2BK)
vh E <
Pa +
(80)
K
Also with probability at least 1 - et
K +
9K
t(11(b a) 02 BK)
Yh E H. Pm[M]] S
Ph
B p* +
(81)
n
where Pf (xx))) and + h(xx))
Lemma Given hypothesis class X + [-b, b) with some universal constant and its VC-dimension dyc(F) <0. Define following sub-root function with B 2 1:
(b(r) 100BEs, R. (+F,)) 110 log
n
Let r" be fized point of (br) so that r" = (6(r),, suppose n 2 dvc(F), we have
13 B3) log(
n
Proof. The proof largely follows from the proof Corollary 3.7 in (Brtlettt al. 2005). We include it here for completeness. Since ff is uniformly bounded by for any r 2 (r)), Corollary 2.2 in Bartlett et al. 2005 implies that with probability at least 1 1 {ʃ E +F pf2 < r) C ( +F pas < 2r). Let & be event. that {f E xF: Pf2 S +F P,S 2r} holds, above implies
Es, (f <r}
<PIE|Ss R R= Pp < PI <T]1
(82)
SEs, Rr, (xF ( 2r b n
38

---

Since r- =)((r), r" satisfies
b+ 11b log
S 100BEs, Rn F E 2r
(83)
Next we leverage Dudley's chaining bound Dudley 2014 to upper bound (*, I 2r*) using integral of covering number. We first bound the covering number of star hull of F. It. follows from (Bartleti et al. 2005 Corollary 3.7 that
(
(
F, S log N22
2 21mm
And covering number (s, F,n)
can bounded using VC-dimension of using Haussler's bound on the covering number (Haussler, 1995 Wellner al. 2013):
log N22 ( F. n) s cidyc log (
where C1 is some universal constant. Now we are ready to apply the chaining bound, it follows from Theorem B.7 Bartlett et al. 2005) that
Bs. (*J P.P 2r")]
22
C2 Es,
Mog.NN(E, +F Fre
vn
C2 Es. m
log
2:
F xl.n
(121+1)) de
(84)
dvc(F)r* log(1/r")
<ez
dvc(F)r* log(n/ log(n/edvc (Fo(F)
<03
n²
n
Where 02 and are some universal constants. B² dF (
Together with Equation 83 one can solve for x*
Lemma 8. Let Sn (( vii} be iidd somple from Data Generalive Process described in Desinition 7 For every >> O, there exist 5 > O such that is n 2 dvc(F) logí )+oR(( sollowing inequalily holds simullaneously for all f E F with (FC < co, with probability at least 5
1(f(z)) #
(x))
(85)
Proof. The proof invokes Lemma in particular, the Equation 81 Let 1 ((f(a) + f((x), be the hesothesis class H in Lemma Since f is a deterministic boolean function, it does not increase the s dvc(F). In particular, we choose the number of points that can be shattered by F. We have o) functional T(-) = E[-] and it easy to verify that
Var(1ff(z) + (x))) S
ff))))
Let (br) 100ER, (xF < r) 11lggn
We have
a
11 log = xr)r)
(R 2 <r} $ ER. <
100ER, (=F. <r}
39

---

Since local Rademacher averages of the star-hull is sub-root function, know for all r", (((r) 2 w0(r)) =
By Equation 81 in Lemma i we have
(((a)) # (x:) = 2 2 ((())) # (x)) + 15r" + log(1/6) 5200
n 8
(86)
Next we bound r A direct application of Lemma
show that
dvc(1 6 log dycllor)
dvc(F) log dvc(F))
n
n
n
The rest of the proof follows from plugging r² in Equation 81 and removing absolute constants,
D More Experimental Results and Details
D.1 Experiment Setting and Implementation Details
Extension multi-class. Our method extends to the multi-class sotting naturally. In the case of K-class classification, our selector loss remains the same while the predictor becomes f'x)) f(x) X + A where A is the K-simplex. Meanwhile, pseudo-informative label becomes 2 [arg maX f(æ:): =.).
multi-class cross entropy loss to train the classifier. The
Hyper-parameters and Neural Network Architectures We list the hyper- parameters and neural network architectures in Table and 6
Table 4: Hyper-parameters used in real-world experiments. Notation follows original paper. Hyper-parant
Bascline
SLNet
a 0.5, À = 32
DeepGambler
0 = in Volatility and = 1.5 in LC and BUS, Pretrain Epoch=10 10 a 0.9, Pretrain Epoch=10
Adaptive
Oneside
J 0.5, Pretrain Epoch=10
ISA
8 30, AT 2, Pretrain Epoch=10 10
Table 5:
CNN Architecture.
Layer Name
Filter Size
Output Size
2d Convolution
3x3
32x28x28
ReLU
1x28x28
2d MaxPool
2x2
32x 32x14x14 14
2d Convolution
3x3
64x 14x14
ReLU
64x14x14
MaxPool
2x2
64x7x7
Linear
600
Drop-out
Linear
120
Linear
10
D.2 More Synthetic Experiments Results.
We present the exact numbers corresponding to Figure 5
We also present experiments results that only use 25% of the dataset below to show the sample efficiency of algorithm. We see all conclusions still hold when we sharply reduce the sample size.
40

---

Table 6: papee--rrameterss used in semi-synthetic experiments. Notation follows original paper.
Dataset
Baseline
Hyper-params
SLNet
asInet 0.5, Aotnet 32 32
DeepGambler
0 1.5, Pretrain p
Adaptive
andaptive = 0.9, Pretrain Epoch=5
MNIST + Fashion
Oneside
# = 0.5, Pretrain Epoch-5
ISA
= 3, AT 10, Pretrain Epoch=5
SLNet
asInet = 0.5, Astnet = 32
DeepGambler
0 2.6, Pretrain Epoch=10
Adaptive
Gadaptine 0.9, Pretrain Epoch=10
SVHN
Oneside
# 0.5, Pretrain Epoch=10
ISA
3 = 10, AT = 1, Pretrain Epoch=10 10
Table 7: g" recovering: AP 1 v.s a with 100% Sample Size
Dataset
Confidencce
ST.Net
DeepCambler
Adaptive
Oneside
ISA
ISA-V2
0.20
1.000 + 0.000
0.982
0.011
1.000 + 0.000
0.807 = 0.027
1.000 = 0..00
1.000 = 0.000
1.000
0.000
0.33
1.000 + 0.000
0.958
0.002
1.000
0.000
0.825
0.027
1.000
4: 0.000
1.000
+ 0.000
1.000
0.000
MNIST + Fashion
0.43
1.000
0.000
0.943
0.002
1.000
0.000
0.855
0.003
1.000
+ 0.000
1.000
+ 0.000
1.000
0.000
0.50
1.000
0.000
0,988
0.003
1,000
0,000
0,829
0.011
1.000
0.000
1.000
0.000
1.000
0.000
0.20
0.984
0.001
0,621
0.256
0.531
0.270
0.984
0.001
0.957
0.007
0.989
0.001
0.982
0.001
SVHN
0,33
0.989 + 0.001
0.868
0.039
0.990 + 0.001
0,980
0.001
0.978
= 0.002
0.992 = 0.000
0,957
0.001
0.43
0.990
1 0.001
0.925
1 0.002
0.990 1 0.000
0.980
+ 0.000
0.982
1 0.001
0.993 = 0.001
0,988
0.1X11
0.50
0.991
+ 0.000
0.914 = 0.053
0.991 + 0.000
0.989 = 0.001
0.983
= 0.002
0.993 = 0.001
0.991 = 0.001
Table 8: g" Recovering: AP v.s. a with 25% Sample Size
v.s.
Dataset
Conficience
SLNet
DeepGambler
Adaptive
Oueside
ISA
ISA- V2
0.20
1,000 + 0.000
0.112 = 0.052
0.985 + 0.019
0.113 = 0.021
0,969 + 0.002
1.000 + 0.000
1.000
= 0.000
0.33
1.000 i 0.000
0.704 + 0.208
0.990 1 0.001
0.362 = 0.011
0.977 1 0.004
1.000 + 0.000
1.000
= 0.000
MNIST Fashion
0.43
1.000
1 0.000
0.881 1 0.179
0.987 1 0.010
0.333 1 0.006
0.976
1 0.007
1.000
0.000
1.000
1 0.000
0.50
1.000
0.000
0.876
0.186
0.991 0.009
0.336
0.006
0,980
0.004
1.000
0.000
1.000
0.000
0.20
0.952
0.005
0.711 + 0.121
0.540 + 0.024
0.909 + 0.013
0.607 + 0.125
0.972
0.003
0.856 + 0.115
SVHN
0.33
0.967
0.001
0.914 + 0.033
0.979 + 0.002
0.950 0.009
0.926 + 0.019
0.981
0.001
0.970 + 0.003
0.43
0.973
0.001
0.931 + 0.021
0.983 0.002
0.967
0.003
0.962 + 0.001
0.984
0.001
0.977 0.001
0.50
0.978 + 0.002
0.917 = 0.006
0.986 + 0.000
0. 0.966- = 0.005
0.965 + 0.001
0.985 + 0.001
0.981 0.001
Table 9: g recovering; AP f v.8 À with 100%
Sample Size
Datasel
A
Confidence
SLNet
DempGenbber
Adaplive
Oueside
ISA
ISA-V2
0.1
0.915 + 0.010
0.599 + 0.168
0,985 + 0.010
0.550 + 0.033
0.862 = 0.033
1.000
0.000
0.999 = 0.001
MNIST + Fashion
0.2
0.986 + 0.001
0.712 + 0.140
0.992 + 0.013
0.566 + 0.018
0.973 + 0.004
1.000
0.000
1.000 + 0.000
0.3
1.000 + 0.000
0.886
0.148
1.000 + 0.000
0.571
0.008
0.908 + 0.000
1.000
0.000
1.000 + 0.000
0.1
0.955 1 0.004
0.641
1 0.164
0.736 1 0.045
0.952 1 0.002
0.944 1 0.001
0.931 1 0.021
0.956 1 0.003
SVHN
0.2
0.978
+ 0.002
0.874 0.029
0.971 + 0.003
0.953 + 0.001
0.972 = 0.001
0.977 0.003
0.978 = 0.002
0.3
0.989
1 0.002
0.949
0.013
1 0.002
0.970 0.008
+ 0.002
0.986
0.002
0.987 0.000
Table 10: g" recovering: AP 1 v.s À with 25% Sample Size
Dataset
A
Confidence
SI.Net.
DeppGambler
Adaptive
Onesirle
ISA
ISA-V2
0.1
0.866 i 0.008
0.507 1 0.011
0.799 i 0.155
0.935 i 0.027
0,581 1 0.017
1.000 # 0.000
0.968 1 0.038
0.2
0.974 + 0.005
0.625 + 0.100
0.917 # 0.015
0.795 + 0.013
0.718 + 0.031
1.000
+ 0.000
1.000 + 0.000
0.3
0.999 + 0.000
0.686 4 0.105
0.972 = 0.010
0.521 + 0.055
0.870 + 0.001
1.000 : 0.000
1.000 + 0.000
0.1
0.883 1 0.009
0.784 1 0.086
0.567
1 0.036
0.795
1 0.015
0.820 1 0.014
0.848 1 0.084
0.892 1 0.008
SVHN
0.2
0.938
+ 0.013
0.916
+ 0.039
0.811
+ 0.108
0.867
+ 0.012
0.913 + 0.011
0.944 + 0.006
0.959
+ 0.006
0.3
0.965
+: 0.002
0.938 1 0.015
0.938 : 0.013
0.901
1 0.029
0.952 1 0.006
0.967 1 0.000
0.969 1 0.004
D.3 Real-World Dataset Description
Oxford realized volatility (Volatility) (Heber et al.] 2009) data set contains 5-min realized volatility of 31 stock indices from 2000 to 2022 and 155107 records in total. We use past volatility and returns as features, and the task here is to predict whether the next day volatility will be higher than current one, making it
41

---

Figure 5: Average Precision (AP) 1 v.8 Different X
25% Samples Size.
Numerical results Table
MNIST
SVHN
<ogg(-AAP)
6
A=0.06
A=0.17
A=0.28
A=0.06
A=0.17
A=0.28
confidence
selectivenet
decpgambler
adaptive
oneside
isa
isav2
binary classification task. We choose data from 2000 to 2020 training set and the rest for the testing (2020 Jan. to 2021 Oct.). This data set is used as example to show our algorithm's possible application in selectively forecasting financial time series.
Breast ultrasound images (BUS) (Al-Dhabyani et al. 2020). BUS contains 780 gray-scale breast ultrasound images among women ages between 25 and years old. These images have average size 500 x 500 pixels and can be categorized into 3 classes (487 benign, 210 malign and 133 healthy). We randomly choose 80% of data training dataset and the rest. 20% for testing. We are going to this dataset an example to show possible application of our algorithm in automatic diagnosis. The machine generate diagnosis result only selected cases and deliver unsure cases to human expert for further investigation.
Lending club(Lending Club, 2007) is a peer-to-peer lending company that matches borrowers with investors through an online platform. The lending club dataset (LC) contains loan data its customers from 2007 to 2018. We compare different version of existing dataset of LC and remove all inconsistent and incomplete records. There are different status of loans record this dataset, we keep types of these record that consist the major part of the dataset (261442 charged off cases, 1035418 fully paid cases and 25757 late cases). We use 20% of the dataset as the testing set. This example shows algorithm can be use to grant loan given on different risk preference.
Table presents the original accuracy given by neural network on each of these real-world data set. For dataset, the neural network without using selection mechanism cannot give reliable inference. In mortgage granting, high risk like this can cause significant loss. medical diagnosis which is healthy issue critical, diagnosis with miss-diagnose rate high as 15% is acceptable. However, if we apply our selective algorithm, see that the risk all dataset sharply reduced. In BUS dataset, perfectly guarantee the diagnosis result empirically for our most confident cases. These evidence are of practical interest.
even almost
Table 11: Real-world Dataset Description
Dataset
Category
Input Feature
Num. Class
Train
Test
Volatility
Time Scries
2
2
143784
9525
BUS
Image
3x324x324
3
624
156
LC
Tabular
1805
3
1058093
264524
D.4 Ablation Study Results
E Illustrative Example for Algorithm 1
42

---

Table 12: DNN Original Risk on Each Dataset
Volatility
BUS
LC
Risk
0.340:10.002
0.152:10.088
0.392-10.001
10
1.0
2.5
Ivverage (racision
Average Proction
Average Procision
0.7
27
0.64
Pretrain=1
Pretrain=b
peta-3.0
AT=1
Pretrain-11
6/1
bela=5.0
24-
aT=5
Rrerain=15
4T-10
Pretrain-2C
30
Training Epoch
Training Epoch
Training Epoch
peta- 3.0
4T=1
Pretrain=1
4T=5
Frttrlln=5
bela=10.0
4T=10
Pretrain=10
Pretrain-15
Selective 1üsk
0.-
Selective Rick
Selective Rick
Pretrain-2C
0.3
35
Taining
Training
Figure 6: Ablation Study. First column shows the result of different choices of 3. Second column shows different choices of AT. Third column shows different choices of pre-training epochs which gives different initial model f())
Upper panel presents results of average precision and bottom panel presents results of selective risk.
43

---

f: Ground-truthh Classification Labels
Uninformative Informative Domain
2: Latent Informative Data Labels
===0
9'=0
Z-0
f=1
g'=1
2-1
=1
Ply" -1)
Pig -1-3
1
Plf 1)-3
PI 11
(b)
(c)
Noisy Samples from DGP
f ERM Output Pf-= 000600
ŽISA Output PG- f³) 009954
7=0
f=0
y=0
y-1
i-1
Pig" =1=3
Pig" =d]=1)=]
Pig 1|===1==3
Plf" =
P(r" 1
PIf 1)=
(d)
Figure 7: Illustration of Algorithm when Ā = 0.4 and a = 0.1. The middle blue horizontal line is the ground truth classifier f The vertical green line is the boundary separates Q2 and Zr. The data generation process follows Definition 1 We use SVM with degree-2 polynomial kernel for both ff and I. a) shows the ground truth classification labels. b) shows the region of D, and Du. c) shows the latent informative label d) shows the realized samples from the noisy data generation process. e) shows the classification result given by ERM. f) shows the classification result given by ISA.
44

---

1.0
0.8
Prīg(x) (x)]
0.4
MNIST: a 0.50
0.2
MNIST: a 0.25
SVHN: a 0.50
SVHN: a 0.25
0.0
0.1
0.2
0.3
0.4
A
Figure Accuracy on recovering g (z).
45


Reviews:
[{"review_id": "LC7CexFlNi", "reviewer": "Reviewer_rVgM", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Overall this is a good paper. The theoretical results provide ample support for the methodology of using this selective learning approach. One can see from the Theorem 2 that a good selector will yield a stronger guarantees for the informative points by leveraging the selector.\n\n\nThe loss function it self used to derive the theoretical algorithm is also intuitive and easy to understand, which I think is a major bonus.\n\n\nThe practical algorithms also appear to yield strong results on both synthetic and real datasets, compared to the baselines. The experiments are also thorough and provide ablations for a better understanding of the choices made for the practical algorithms.\n\n\nOne potential downside is that it is not clear how well the method fairs at extreme values of $\\alpha$.\n\n\nOther than this I don’t have really any major weaknesses discuss, but I have pointed out some changes that would better help the presentation of the paper listed below.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies the problem of learning from both informative and uniofrmative data. In particular, the setting supposes there are distributions of informative (potentially low noise) and uninformative (potentially high noise) data. The authors propose a selective learning method that attempts to distinguish the informative data points from the uninformative ones. The resulting learner can then ‘condition’ on the informative data points and potentially benefit from an enhanced sample complexity bound. The main contributions are as follows:\n\nThe theoretical selective sampling algorithm is proposed, which is iterative where each iteration consists of two stages.\n\nTheoretical results are derived for a theoretical algorithm, which justifies the methodology.\n\nTwo practical versions of the algorithms are proposed along with a number of heuristics. Experiments with the practical algorithms show that they are effective in learning to distinguish and use the informative data.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The paper states that the experiments are run with varying $\\alpha \\in \\{ 1, .75, .5, .25 \\}$. However, I don’t see where this is represented in the tables. I would advise making this more clear.\n\nThe titles like ‘purely uninformative data’ do not match how the datasets are described in the text which makes it difficult to quickly understand the results.\n\nI do not think the results are best represented as a series of long tables. Once again, it is difficult to get a grasp for how the different methods compare without spending a great deal of time scrutinizing the numbers. Many of the tables could be replaced with bar graphs.\n\nFor Table 5, is there some way to understand what would be the naive performance of simply doing ERM? From my understanding, Confidence is also doing some selection, but just in a very naive way.", "broader_impact_concerns": "No concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "VcSn6LLGNB", "reviewer": "Reviewer_VdEU", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n- Relevant problem area with good motivation\n- Mostly well written and organized\n- Through ablation studies\n\n### Weaknesses\nRelated Work\n- Brief comparison to [1-2] which should be discussed in more detail. The paper describes differences in motivation, but are any bounds directly comparable to Theorems 1-2?\n\nExperiments\n- The experiments would be improved if the choice of average precision as a main metric was better motivated\n\nClarity\n- The term \"heuristic\" often implies a simple rule that is adopted without proof (e.g. Baseline 5 on page 8). Since ISA and ISA-V2 originate from the guarantees of Theorem 2, in this case the term \"practical algorithm\" is more appropriate for these algorithms.\n- \"subset-risk minimization\" is mentioned without definition\n- Notation alternates between $\\Omega_D$ and $\\Omega$\n- Learning rate $\\eta$ is mentioned in Algorithm 1 but not used\n- \"remains an opened problem\" should be \"remains an open problem\"\n- Some important details are only mentioned in the Appendix (see below)\n\n[1] Cortes et al. Learning with rejection. In ICALT, pp. 67–82, 2016.\n\n[2] Yonatan Geifman and Ran El-Yaniv. Selectivenet: A deep neural network with an integrated reject option. In ICML, pp. 2151–2159, 2019.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies the problem of classification where a fraction of the data is uninformative, i.e. likely to have randomly assigned labels. The authors prove theoretical bounds on the performance of a data selection model $g$ that depends on the quality of the predictor $f$. This leads to a natural alternating minimization approach to jointly optimize $f$ and $g$. Empirical results show that a relaxed version of the proposed algorithm outperforms baselines on MNIST + Fashion and SVHN datasets.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "### Critical\n- Compare results to those stated in [1-2] if applicable, i.e. expand on statements such as Remarks 4-5 and move them to the main paper\n- Add informal statement about the range of $\\beta$ to Section 4\n- Motivate the use of average precision as a performance metric rather than recall/F1-score/AUC. Is $\\beta$ tuned to maximize precision?\n\n\n### Non-critical\n- Show how performance varies with different quality of the initial $\\hat{f}^0$\n- Describe the main technical challenges to proving convergence guarantees for ISA and ISA-V2 (applying Theorem 2 to Algorithm 1 and relaxations 1-4)\n- Compare to semi-supervised learning methods, i.e. discard only the labels of uninformative data\n\n[1] Cortes et al. Learning with rejection. In ICALT, pp. 67–82, 2016.\n\n[2] Yonatan Geifman and Ran El-Yaniv. Selectivenet: A deep neural network with an integrated reject option. In ICML, pp. 2151–2159, 2019.", "broader_impact_concerns": "No concerns about broader impact", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "6qkNTvkTPe", "reviewer": "Reviewer_UPfk", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The paper has a comprehensive presentation of the problem at hand; it starts off with a reasonable motivation intro, then formulates it and proves theoretical guarantees for their framework; finally using several empirical analyses, they complement their theoretical assessments.\n2. The paper proposes a practical algorithm for their minimax framework; this is usually lacking in minimax optimal guarantees.\n\nWeakness:\n1. The presentation could improve:\n1.1. The psudeo-informative loss is defined as $\\mathbb{1}\\{f(x)\\neq y\\}$ which lies in $\\{0,1\\}$, as a surrogate for $z$ which is $\\in\\{-1, 1\\}$. That causes confusion. Also, later on similar expressions are defined with $=$ instead of  $\\neq$, like line 4 of Algorithm 1.\n\n1.2. $\\beta$ is not clearly defined at Eq. (5) and is later discussed.\n\n2. The claims are not well-supported: the main proposed and analyzed algorithm, Algorithm 1, is missing from the experiments in Sections 7.1-7.3 and is represented with its relaxations, ISA and ISA-V2. This while Section 7.4 shows Algorithm 1 performs drastically worse than its relaxations.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper study a high noise-to-signal setting where the data is partitioned into two groups: informative data and the uninformative data. They propose a new selective training loss for this setting using two models: selector and predictor. They provide minimax and joint risk bounds for their framework. An iterative algorithm is proposed that solves the problem practically. Experimental results compare their algorithms to the benchmarks on synthetic and real-world data.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Please add the corrections required to answer / clarify the \"weaknesses\" items above.\n\n2. Few writing mistakes could be fixed: the missing space in line 3 of Section 7.1, wrong style for the citation in the first line of page 9, the missing space in the \"baselines\" part of Section 7, between \"OnesideGangrade et al. ...\".", "broader_impact_concerns": "Not needed.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper considers learning (mostly classification) in a setting in which a fraction of the data is uninformative, i.e. on that data the observed label is random noise unrelated to the true target. The method proposes simultaneously learning a "selector" which tries to identify which data is informative and the "classifier" which does the classification. The proposed algorithm is an alternate optimization method for learning these two functions. There is an algorithm for which there are guarantees proved under some conditions. There is also an easier to implement algorithm presented, which is more of a  heuristic, and relies on estimating whether or not a data point is informative by evaluating whether or not its predicted label is correct, and then these pseudo-informative labels are used to update the selector. The classifier is trained by weighting the data using the selector.  There are experiments on MNIST and SVHN which demostrate the efficacy of the proposed methods.
Decision: Accept (poster)