On the interplay between data structure and loss function in classification problems

1 Introduction . Classical wisdom teaches us that a learning model should have just the right number of parameters to learn from a dataset without overfitting it . However , recent years have seen the emergence of massively over-parametrized models which manage to generalize well on high-dimensional tasks [ 1 , 2 ] , somehow avoiding both the curse of dimensionality and the pitfall of overfitting . This generalization capacity in the over-parametrized regime continues to puzzle rigorous understanding , in particular for deep neural networks [ 3 , 4 , 5 ] , despite their remarkable achievements over the past decade [ 6 , 7 , 8 , 9 ] . Various works have given evidence of a double descent curve [ 5 , 10 , 11 , 12 , 13 ] , whereby the test error first decreases as the number of parameters increases , then peaks , then decreases again monotonically . The overfitting peak occurs at the interpolation threshold where training error vanishes , a well-studied phenomenon in the statistical physics literature [ 14 , 15 , 16 , 17 , 18 ] . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Although the underlying structure of data plays a major role in the generalization ability of overparametrized models [ 19 ] , it has been little studied from a theoretical point of view . The first aspect of data structure is the distribution of the inputs : MNIST and CIFAR10 have the same number of classes and images , yet generalization is harder for CIFAR10 because the images are more complex than handwritten numbers . The second aspect is the rule between inputs and outputs : a random labelling of CIFAR10 can be learned by a neural network but offers no possibility of generalization [ 3 ] . Characterizing the structure of real-world data involves studying the interplay between these two aspects . In this direction , simple and interpretable models of structured data can prove useful to understand what underlies the behavior of generalization . In this work , we present a simple model of learning and its analytical solution which simultaneously enables us to : ( i ) study the effect of overparametrization for a given task ; ( ii ) analyze both regression and classification tasks ; ( iii ) control the structure of the inputs and their relationship with the labels . To satisfy ( i ) , we need to disentangle the input dimension from the number of parameters in the learning model , which is impossible for the linear models often studied in litterature . We instead consider a random feature model [ 20 ] , which was shown to exhibit double descent by [ 21 ] . To satisfy ( ii ) , we follow the lines of [ 22 ] and use an approach from statistical physics , enabling to generalize the result of [ 21 ] to any convex loss function . Finally , to satisfy ( iii ) , we introduce a block model in which the input space is subdivided into various subspaces with different variances ( saliencies ) and different correlation ( alignments ) with the labels ( see Fig . 1 ) . This model was recently studied for linear regression under the name of strong and weak features model [ 23 ] . Our analytical solution uses the replica method from statistical physics [ 24 ] . Albeit non-rigorous in a strict mathematical sense , this method is considered exact in theoretical physics and has been influential to statistical learning theory since the 80s , see [ 25 ] and [ 26 ] for recent reviews . Several of their conjectures have been proven exact in the last decade [ 27 , 28 , 29 , 30 ] , including in settings very close to the considered model as discussed further at the end of Section 3 [ 31 , 32 ] . Contributions Our main analytical contribution , presented in Section 3 , is the expression of the train and test errors of a random feature model trained on data generated by the strong and weak features model . Our analysis is valid in the high-dimensional limit both for regression and classification tasks , although we focus here on the latter . We confirm that the asymptotic prediction matches simulations at finite size ( see Fig . 2 and 4 ) . We leverage this result to present a thorough analytical study of how the data structure and the loss function interplay to shape the generalization error , and in particular how they affect the double descent curve ( Section 4 ) . We highlight behavioral differences between square loss and logistic loss , in particular the fact that logistic loss generalizes better for easy tasks ( see Fig . 3 ) . We validate our insights via controlled experiments on the MNIST and CIFAR10 datasets described in the Section 5 . Reproducibility The code to reproduce our experiments is available at https : //github.com/ sdascoli/data-structure . 2 Related Work . Although most theoretical studies of generalization focus on structureless data , a few exceptions exist . The strong and weak features model has recently been studied in the context of least-squares regression , both empirically [ 33 ] and theoretically [ 34 , 35 , 23 ] . Several intriguing observations emerge in strongly anisotropic setup : ( i ) several overfitting peaks can be seen [ 33 , 36 ] ; ( ii ) the optimal ridge regularization parameter can become negative [ 35 , 37 ] , as it becomes helpful to encourage the weights to have very different magnitudes ; ( iii ) extra features acting as pure noise can play a beneficial role by inducing some implicit regularization [ 23 ] . The strong and weak features model also includes the setup studied in [ 38 ] , called the spiked covariates model . The latter involves a small block of size D⌘ ( ⌘ < 1 ) and a large “ junk ” block with no correlation with the labels . The question is then : can kernel methods learn to discard the junk features , hence “ beat the curse of dimensionality ” in the way neural networks do ? The answer was shown to depend on the strength of the junk features : when the variance of these features is small , they are not problematic , and the kernel method ignores them , effectively learning a task of effective dimensionality D⌘ ⌧ D. More general models for the structure of data were considered in other works . In the special case of random Fourier Feature regression , [ 39 ] derived the train and test error for a general input distribution . [ 40 ] achieved a similar result in the non-parametric setting of kernel regression , which can be viewed as the limiting case of random feature regression when the number of random features P goes to infinity . The few works studying classification analytically have mostly focused on linear models , trained on linearly separable data [ 41 , 42 , 43 , 44 ] or gaussian mixtures [ 45 , 46 ] . One of the challenges in classification ( in comparison to regression ) is the large set of available loss functions [ 47 , 48 ] . In the context of random feature models , [ 22 ] uses tools from statistical mechanics to derive the generalization loss of random features model for any loss function with i.i.d . Gaussian input . More recently , [ 32 ] shows that this framework could extend to more complex data distributions and learned feature maps provided that key population covariances are estimated by Monte Carlo methods . Our paper crucially builds on these contributions , by deriving a fully analytical analysis for a simple interpretable model of data structure , while also analyzing the effect of label flipping . We also note that a few recent works investigate the roles of the loss and the structure in data in more realistic setups where theoretically robust results are harder to obtain . Several works have studied the low intrinsic dimensionality of real-world data distributions and how it impacts sample complexity for supervised tasks [ 49 , 50 , 19 , 51 ] . The impact of the loss function is also an active research area : [ 52 , 53 ] show that square loss can perform equally or better than the ubiquitous cross-entropy loss in realistic multi-class classification problems , if one rescales the weight of the correct class to emphasize its importance . 3 A solvable model of data structure . In this work , we focus on the random features model1 introduced in [ 20 ] . Using tools from statistical physics , we derive the generalization error of a teacher-student task on the strong and weak features model of structured data . 1Note that this model is akin to the so-called lazy learning regime of neural networks where the weights stay close to their initial value [ 54 ] : assuming f✓0 =0 , we have f✓ ( x ) ⇡ r✓f✓ ( x ) |✓=✓0 · ( ✓ ✓0 ) [ 55 ] . In other words , lazy learning amounts to a linear fitting problem with a random feature vector r✓f✓ ( x ) |✓=✓0 . Random feature model The random features model can be viewed as a two-layer neural network ( see Fig . 1 ) whose first layer is a fixed random matrix containing P random feature vectors { Fi 2 RD } i=1 ... P acting on inputs xµ 2 RD : ŷµ = PX i=1 wi ✓ Fi · xµp D ◆ , ( 1 ) where ( · ) is a pointwise activation function and wi 2 R are the second layer weights . Elements of F are drawn i.i.d from N ( 0 , 1 ) . The second layer weights , i.e . the elements of w 2 RP , are trained by minimizing an ` 2-regularized loss on N training examples { xµ 2 RD } µ=1 ... N : ŵ = argmin w ✏t ( w ) + 2 kwk22 , ✏t ( w ) = NX µ=1 ` ( yµ , ŷµ ) , ( 2 ) where ` denotes the logistic loss ` ( y , ŷ ) = log ( 1+e yŷ ) . The target labels are given by a probabilistic teacher y ⇠ Pt ( y| · x ) corresponding to the sign of a linear function possibly corrupted by label flipping : yµ = ⌘µ sign ✓ · xµp D ◆ , ⌘µ = ⇢ 1 with probability 1 1 with probability . ( 3 ) The generalization error is computed as the 0-1 loss , ✏g = Ex , y ⇥ sign ( ŷ ( x ) ) , y ( x ) ⇤ . ( 4 ) Strong and weak features model To impose structure on the input space , we introduce a blockstructured covariance matrix from which the elements of the inputs x 2 RD and the teacher 2 RD are sampled as : x ⇠ N ( 0 , ⌃x ) , ⇠ N ( 0 , ⌃ ) , ⌃x = 2 64 x,1I 1D 0 0 0 x,2I 2D 0 0 0 . . . 3 75 , ⌃ = 2 64 ,1I 1D 0 0 0 ,2I 2D 0 0 0 . . . 3 75 . Our result presented in the rest of Section 3 is valid for an arbitrary number of blocks . In Section 4 we will focus for interpretability on the special case where we only have two blocks of sizes 1D and 2D , with 1 + 2 = 1 . We will typically be interested in the strongly anisotropic setup where the first subspace is much smaller ( 1 ⌧ 1 ) , but potentially has higher saliency rx = x,1/ x,2 1 ( see Fig . 1 ) . Main analytical result Using the replica method from statistical physics [ 24 ] and the Gaussian Equivalence Theorem ( GET ) [ 56 , 21 , 30 , 31 ] , we derive the generalization and training errors in the high-dimensional limit where D , N and P ! 1 with fixed ratios . The asymptotic generalization and training errors are given by lim N ! 1 ✏g = 1 ⇡ cos 1 ✓ Mp ⇢Q ◆ , ( 5 ) lim N ! 1 ✏t = E⇠ Z R dy h Z0 ⇣ y , ⇠M/ p Q , ⇢ M2/Q ⌘ ` ( y , ⌘ ( y , p Q⇠ , V ) i ( 6 ) with the proximal operator ⌘ ( y , a , b ) = arg min x ( x a ) 2/ ( 2b ) + ` ( x , y ) , the random variable ⇠ ⇠ N ( 0 , 1 ) , the functional Z0 ( y , a , b ) = R R dx N ( x ; a , b ) Pt ( y|x ) and the scalars ⇢ = X i i , i x , i , M = 1 X i x , ims , i , Q = 2 1 X i x , iqs , i + 2 ? qw , V = 2 1 X i x , ivs , i + 2 ? vw . The parameters 1 , ? are related to the activation function : denoting r = P i i x , i and ⇠ ⇠ N ( 0 , r ) , one has 1 = 1 r E⇠ [ ⇠ ( ⇠ ) ] ] , ? = q E⇠ [ ( ⇠ ) 2 ] r21 . ( 7 ) Besides these constants and the ones defining the data structure ( i , , i , x , i ) , the key ingredients to obtain asymptotic errors are the so-called order parameters ms , qs , qw , vs and vw . They correspond to the high-dimensional limit of the following expectations and variances ( denoted by V ) : ms , i = lim D ! 1 1 D EP [ si · i ] , qs , i = lim D ! 1 1 D EP [ si · si ] , qw = lim P ! 1 1 P EP [ ŵ · ŵ ] , vs , i = lim D ! 1 1 D VP [ si · si ] , vw = lim P ! 1 1 P VP [ ŵ · ŵ ] , where s = 1p P Fŵ 2 RD and si , i 2 R iD denote the orthogonal projections of s and onto subspace i 2 { 1 , 2 } and P denotes the joint distribution of all random quantities in the problem ( the teacher weights , the random features and the training data ) . Intuitively , ⇢ is the variance of the outputs of the teacher , Q is the variance of the outputs of the student , and M is their covariance . The generalization error is given by the “ angle ” between the teacher and the student , as expressed by Eq . 5 . The order parameters allowing to obtain Q and M are one of the outputs of the replica computation deferred to SM B . They are obtained by solving a set of non-linear saddle-point equations ( see Section B.5 of the SM ) . Our framework is valid for any convex loss function , although the replica equations need to be evaluated numerically in the general case . In the case of the square loss however , some simplifications arise , e.g . an explicit expression for the training error can be obtained from the order parameters ( see Section B.6 of the SM ) . Steps and validity of the replica analysis The necessary steps of the derivation are detailed in SM B . In particular , ( i ) we obtain an anistropic extension of the GET , ( ii ) conduct random matrix analysis for block matrices and finally ( iii ) derive the analytical saddle-point equations which yield the values of the order parameters . Our result generalizes the strategy of [ 22 ] from isotropic to anisotropic data and additionally covers the effect of label flipping . Our result is also related to [ 32 ] which establishes rigorously the replica prediction in related learning problems . A rigorous proof of our replica results is within reach : it requires a small extension of [ 31 ] ( to prove the anisotropic GET derived in Section B.2 of the SM ) combined with the recent results of [ 32 ] . Moreover , results in the following section show perfect agreement with numerical experiments .

Summary: This paper used the replica method to derive the high dimensional asymptotics of the test error of random features model in the strong and weak features model with square loss and logistic loss. The observations drawn from plotting the analytical prediction are the following: 1. When the parameters vector of the target function is aligned with the covariance structure of the covariates, the generalization error is smaller (given fixed N/D, P/D, lambda, loss function). 2. In the aligned settings, the test error of using the logistic loss is smaller than that of using the square loss (in a specific setting). This paper also performed experiments on parity MNIST and CIFAR10(2), and the experimental result coincides with the results with idealized assumptions.