Experimental Design for Overparameterized Learning with Application to Single Shot Deep Active Learning

1 INTRODUCTION . The impressive performance exhibited by modern machine learning models hinges on the ability to train the aforementioned models on a very large amounts of labeled data . In practice , in many real world scenarios , even when raw data exists aplenty , acquiring labels might prove challenging and/or expensive . This severely limits the ability to deploy machine learning capabilities in real world applications . This bottleneck has been recognized early on , and methods to alleviate it have been suggested . Most relevant for our work is the large body of research on active learning or optimal experimental design , which aims at selecting data point to be labeled so to maximally inform the learning process . Disappointedly , active learning techniques seem to deliver mostly lukewarm benefits in the context of deep learning . One possible reason why experimental design has so far failed to make an impact in the context of deep learning is that such models are overparameterized , and oftentimes are trained to be interpolative ( Zhang et al. , 2017 ) , i.e. , they are trained so that a perfect fit of the training data is found . This raises a conundrum : the classical perspective on statistical learning theory is that overfitting should be avoided since there is a tradeoff between the fit and complexity of the model . This conundrum is exemplified by the double descent phenomena ( Belkin et al. , 2019b ; Bartlett et al. , 2020 ) , namely when fixing the model size and increasing the amount of training data , the predictive performance initially goes down , and then starts to go up , exploding when the amount of training data approaches the model complexity , and then starts to descend again . This runs counter to statistical intuition which says that more data implies better learning . Indeed , when using interpolative models , more data can hurt ( Nakkiran et al. , 2020a ) ! This phenomena is exemplified in the curve labeled “ Random Selection ” in Figure 1 . Figure 1 explores the predictive performance of various designs when learning a linear regression model and varying the amount of training data with responses . The fact that more data can hurt further motivates experimental design in the interpolative regime . Presumably , if data is carefully curated , more data should never hurt . Unfortunately , classical optimal experimental design focuses on the underparameterized ( and thus , noninterpolative ) case . As such , the theory reported in the literature is often not applicable in the interpolative regime . As our analysis shows ( see Section 3 ) , the prediction error of interpolative models can either be bias dominated ( the first descent phase , i.e. , when training size is very small compared to the number of parameters ) , variance dominated ( near equality of size and parameters ) or of mixed nature . However , properly trained underparameterized models tend to have prediction error which is variance dominated , so classical experimental design focuses on variance reduction . As such , naively using classical optimality criteria , such as V-optimality ( the one most relevant for generalization error ) or others , in the context of interpolation , tends to produce poor results when prediction error is bias dominated or of mixed nature . This is exemplified in the curve labeled “ Classical OED ” in Figure 1 . The goal of this paper is to understand these regimes , and to propose an experimental design strategy that is well suited for overparameterized models . Like many recent work that attempt to understand the double descent phenomena by analyzing underdetermined linear regression , we too use a simple linear regression model in our analysis of experimental design in the overparameterized case ( however , we also consider kernel ridge regression , not only linear interpolative models ) . We believe that understanding experimental design in the overparameterized linear regression case is a prelude to designing effective design algorithms for deep learning . Indeed , recent theoretical results showed a deep connection between deep learning and kernel learning via the so-called Neural Tangent Kernel ( Jacot et al. , 2018 ; Arora et al. , 2019a ; Lee et al. , 2019 ) . Based on this connection , and as a proof-of-concept , we propose a new algorithm for single shot deep active learning . Let us now summarize our contributions : • We analyze the prediction error of learning overparameterized linear models for a given fixed design , revealing three possible regimes that call for different design criteria : bias dominated , variance dominated , and mixed nature . We also reveal an interesting connection between overparameterized experimental design and the column subset selection problem ( Boutsidis et al. , 2009 ) , transductive experimental design ( Yu et al. , 2006 ) , and coresets ( Sener & Savarese , 2018 ) . We also extend our approach to kernel ridge regression . • We propose a novel greedy algorithm for finding designs for overparameterized linear models . As exemplified in the curve labeled “ Overparameterized OED ” , our algorithm is sometimes able to mitigate the double descent phenomena , while still performing better than classical OED ( though no formal proof of this fact is provided ) . • We show how our algorithm can also be applied for kernel ridge regression , and report experiments which show that when the number of parameters is in a sense infinite , our algorithm is able to find designs that are better than state of the art . • We propose a new algorithm for single shot deep active learning , a scaracly treated problem so far , and demonstrate its effectiveness on MNIST . Related Work . The phenomena of benign overfitting and double descent was firstly recognized in DNNs ( Zhang et al. , 2017 ) , and later discussed and analyzed in the context of linear models ( Zhang et al. , 2017 ; Belkin et al. , 2018 ; 2019a ; b ; Bartlett et al. , 2020 ) . Recently there is also a growing interest in the related phenomena of “ more data can hurt ” ( Nakkiran et al. , 2020a ; Nakkiran , 2019 ; Nakkiran et al. , 2020b ; Loog et al. , 2019 ) . A complementary work discussed the need to consider zero or negative regularization coefficient for large real life linear models ( Kobak et al. , 2020 ) . Experimental design is an well established paradigm in statistics , extensively covered in the literature for the linear case ( Pukelsheim , 2006 ) and the non linear case ( Pronzato & Pázman , 2013 ) . The application of it to pool based active learning with batch acquisitions was explored by Yu et al . ( 2006 ) for linear models and by Hoi et al . ( 2006 ) for logistic regression . It was also proposed in the context of deep learning ( Sourati et al. , 2018 ) . Another related line of work is recent work by Haber and Horesh on experimental design for ill-posed inverse problems ( Haber et al. , 2008 ; 2012 ; Horesh et al. , 2010 ) . Active learning in the context of overparameterized learning was explored by Karzand & Nowak ( 2020 ) , however their approach differs from ours significantly since it is based on artificially completing the labels using a minimax approach . I the context of Laplacian regularized Least Squares ( LapRLS ) , which is a generalization of ridge regression , Gu et al . ( 2012 ) showed rigorously that Yu et al . ( 2006 ) criterion is justified as a bound for both the bias and variance components of the expected error . We farther show that this bound is in some sense tight only if the parameter norm is oneand the noise variance equals the l2 penalty coefficient . In addition we postulate and show experimentally that in the overparameterized case using a bias dominant criterion is preferable . Another case in which the bias term idoes not vanish is when the model is misspecified . For linear and generalized linear models this case has been tackled with reweighing of the loss function . A popular modern approach for pool based active learning with batch acquisition is coresets ( Sener & Savarese , 2018 ; Geifman & El-Yaniv , 2017 ; Ash et al. , 2019 ; Pinsler et al. , 2019 ) . This approach has been used in the context of active learning for DNNs . 2 UNDERPARAMETERIZED V-OPTIMAL EXPERIMENTAL DESIGN . Consider a noisy linear response model y = xTw + , where ∼ N ( 0 , σ2 ) and w ∈ Rd and assume we are given with some data points x1 , . . . , xn , for which we obtained independent responses , yi = x T iw + i . Consider the underparameterized case , i.e . n ≥ d , and furthermore assume that the set { x1 , . . . , xn } contains at least d independent vectors . The best linear unbiased estimator ŵ of w according to the Gauss-Markov theorem is given by : ŵ = arg minw ‖Xw − y‖22 = X +y where X ∈ Rn×d is a matrix whose rows are x1 , . . . , xn , y = [ y1 . . . yn ] T ∈ Rn and X+ is the Moore-Pensrose pseudoinverse of X . It is well known that ŵ −w is a normal random vector with zero mean and covariance matrix σ2M−1 , where M = XTX is the Fisher information matrix . This implies that ŷ ( x ) − y ( x ) is also a normal variable with zero mean and variance equal to σ2xTM−1x . Assume also that x comes from a distribution ρ . With that we can further define the excess risk R ( ŵ ) = Ex∼ρ [ ( xTw − xTŵ ) 2 ] and its expectation : E [ R ( ŵ ) ] = Ex∼ρ [ Var [ y ( x ) − ŷ ( x ) ] ] = Ex∼ρ [ σ2xTM−1x ] = Tr ( σ2M−1Cρ ) ( 1 ) where Cρ is the uncentered second moment matrix of ρ : Cρ : = Ex∼ρ [ xxT ] . Eq . ( 1 ) motivates the so-called V-optimal design criterion : select the dataset x1 , . . . , xn so that ϕ ( M ) : = Tr ( M−1Cρ ) is minimized ( if we do not have access to Cρ then it is possible to estimate it by drawing samples from ρ ) . In doing so , we are trying to minimize the expected ( with respect to the noise ) average ( with respect to the data x ) prediction variance , since the risk is composed solely from it ( due to the fact that the estimator is unbiased ) . As we shall see , this is in contrast with the overparameterized case , in which the estimator is biased . V-optimality is only one instance of various statistical criteria used in experimental design . In general experimental design , the focus is on minimizing a preselected criteria ϕ ( M ) ( Pukelsheim , 2006 ) . For example in D-optimal design , ϕ ( M ) = det ( M−1 ) and in A-optimal design ϕ ( M ) = Tr ( M−1 ) . However , since minimizing the V-optimality criterion corresponds to minimizing the risk , it is more appropriate when assessing the predictive performance of machine learning models .

Summary: In this paper, the authors develop a data selection scheme aimed to minimize a notion of Bayes excess risk for overparametrized linear models. The excess Bayes risk is the expected squared error between the prediction and the target. The authors note that solutions such as V-optimality exist for the underparametrized cases (linear regression), and offer extensions to ridge regression. After the development of a greedy schemes and a tentative extension to deep learning models, the authors show that their selection scheme can outperform random selection on MNIST with a specific model.