Takeuchi's Information Criteria as Generalization Measures for DNNs Close to NTK Regime

1 INTRODUCTION . Deep neural networks ( DNNs ) have been exhibiting great generalization abilities in many applications , but the mechanism of the generalization has not been fully understood yet ( Neyshabur et al. , 2014 ; Zhang et al. , 2016 ; Recht et al. , 2019 ) . Establishing a reliable generalization measure is an important research topic for generating a good model from limited data resources , including an application of hyperparameter search . Many attempts ( Arora et al. , 2018 ; Wei & Ma , 2019 ; Neyshabur et al. , 2018 ) have been taken to better understand the generalization phenomenon in deep learning models from theoretical points of view . From empirical points of view , there have been intensive studies ( Keskar et al. , 2016 ; Liang et al. , 2019 ; Bartlett et al. , 2017 ) in search of learning conditions that likely yield high model performance . Work by ( Jiang et al. , 2019 ) indicated that a measure that includes both hessian H ( θ ) 1 and covariance C ( θ ) defined from the loss and the network parameters θ near a local minimum may potentially show good correlation with generalization performance . Another study indicated that use of only a single measure , either HessianH ( θ ) or covariance C ( θ ) , fails to capture the generalization performance ( Novak et al. , 2018a ) . The generalization gap inherently stems from a discrepancy between the empirical and the true data distribution . A minimizer of the empirical loss will be affected by the noise due to a finite number of samples and by the form of the loss landscape near the minimum . The former can be characterized as noise ( C ( θ ) ) and the latter as curvature ( H ( θ ) ) . Taking these findings into account , we sought to model the generalization gap , then found that a classical information criterion called Takeuchi ’ s Information Criteria ( TIC ) ( Takeuchi , 1976 ) ex- 1H ( θ ) andC ( θ ) are defined in equation 2 presses generalization gap in the neural tangent kernel ( NTK ) regime . TIC has the following form TIC ( θ ) ︸ ︷︷ ︸ Information Criteria = −Ep̂ [ l ( y , f ( x , θ ) ) ] ︸ ︷︷ ︸ Mean Empirical Error + Tr ( H ( θ ) −1C ( θ ) ) ︸ ︷︷ ︸ Estimated Bias Term , ( 1 ) where f is a smooth function over θ ∈ Rd with input x and target y , and l is the negative loglikelihood , also denoted as loss function . The first term on the right-hand side is the log-likelihood , which takes the expectation over an empirical data distribution ( xi , yi ) ∼ p̂ . For the later discussion , we use θ̂ minimizes the empirical loss ; i.e. , θ̂ = arg minθEp̂ [ l ( y , f ( x , θ ) ) ] , and θ ∗ as the parameters that maximizes the likelihood with respect to the true data distribution ( x , y ) ∼ p ; i.e. , θ∗ = arg minθEp [ l ( y , f ( x , θ ) ) ] . For a DNN of practical size , exact computations of the matrices H ( θ ) , C ( θ ) are nearly infeasible due to large dimensionality . To make the computation feasible , we adopted a strategy of using shared components of the matrix to estimate TIC with fewer computations , based on a relationship among matrices such as H ( θ ) , C ( θ ) , and Fisher Information Matrix F ( θ ) . To further reduce the computational costs for the bias term , we examined methods using approximations and lower bounds so that TIC estimations for DNNs of practical sizes are feasible . In this work , we make the following contributions : • We provide empirical and theoretical evidence that TIC is highly correlated with generalization gap of DNNs that are close to NTK regime , despite the fact that TIC is not originally designed for singular model such as general DNNs . • We conduct comprehensive experiments in which more than 5,000 models , including ones close to NTK regime , with totally 12 DNN architectures , 4 datasets and 15 training settings are trained , and corresponding TICs are estimated with approximation techniques , to clarify conditions that TIC can well explain generalization gaps . • We use TIC as a threshold for pruning poorly performing trial models during hyperparameter optimization ( HPO ) and show that it can successfully prevent promising candidates from being pruned prematurely . 2 GENERALIZATION MEASURES . Generalization measures measure the generalization ability of statistical models . Typically , the generalization gap , which is defined as the difference between training loss and validation loss , is used to quantify the generalization ability . 2.1 WHICH GENERALIZATION MEASURE IS PROSPECTIVE ? . To answer this question , before demonstrating the effectiveness of TIC , we highlight the development of research in this area and the motivation behind this work . For understanding generalization behavior , there are two major approaches by quantifying generalization bounds and complexity measures . Approach of quantifying generalization bounds is pursued by theoretical groups to prove the bound of the generalization gap ( Dziugaite & Roy , 2017 ) . Although tight bounds can be proven , they are often based on assumptions that do not apply to practical DNN settings . In addition , no bounds have been shown to describe the performance of the current DNNs to a satisfactory level . On the other hand , approach of quantifying complexity measures , which do not necessarily certify bounds , follows the principle of Occam ’ s razor and evaluates the complexity of the model . Theoretically motivated complexity measures , including VC-dimension ( Vapnik & Chervonenkis , 2015 ) , PAC-Bayesian framework ( McAllester , 1999 ) , the norm of parameters ( Neyshabur et al. , 2015 ) , are often discussed as significant components of generalization bounds , and a monotonic relationship between complexity measures and generalizations is mathematically established . In contrast , empirically motivated generalization measures , such as sharpness ( Keskar et al. , 2016 ) , are justified by experiments and observations . In particular , for DNNs , Jiang et al . ( 2019 ) have conducted exhaustive experiments to evaluate the effectiveness of generalization measures for three groups : norm-based measure , sharpness-based measure , and noise-based measure . • Norm-based measure : |θ| . Most of the proposed norm-based measures are based on the FisherRao Metric ( Liang et al. , 2019 ) , which does not capture generalization well . In particular , it has been reported that spectral complexity such as product of spectral norms of the layers ( Bartlett et al. , 2017 ) has a strong negative correlation with generalization . It is impossible to explain the success of DNN models with huge parameter sizes in recent years with these metrics . • Sharpness-based measure : H ( θ ) . Sharpness-based metrics , such as sharp minima and flat minima ( Keskar et al. , 2016 ) and PAC-Bayesian framework ( McAllester , 1999 ) , are not only associated with intuitive understanding but also empirically show a strong correlation with the generalization gap . However , some model architectures are known to show poor correlation ( Dinh et al. , 2017 ) . • Noise-based measure : C ( θ ) . Experimental results show that generalization measure based on gradient has potential ( Jiang et al. , 2019 ) . In particular , in their experiments , they observe that the variance of the gradient captures Sharpness , but they suggest that this is not a good generalization measure depending on the architecture of the model . These results suggest that studying generalization measures that can be estimated using H ( θ ) and C ( θ ) is prospective . However , since the combination ofH ( θ ) andC ( θ ) seen in TIC is not feasible to compute for practical DNN settings , so it was outside the scope of ( Jiang et al. , 2019 ) . 2.2 INFORMATION MATRIX : ELEMENTS OF GENERALIZATION MEASURES . Previous research has highlighted information matrices such as H ( θ ) and C ( θ ) in generalization measures in DNNs . Thomas et al . ( 2020 ) ; Kunstner et al . ( 2019 ) remarked that these matrices are often confused and misused , for example in the field of optimization , leading to wrong conclusions , even though these matrices play an essential role in the study of DNNs , especially in optimization ( Amari et al. , 2020 ; Martens & Grosse , 2015a ) , understanding implicit regularization in SGD ( Wen et al. , 2019 ; Zhu et al. , 2019 ) , and Bayesian inference ( Zhang et al. , 2018 ) . Before discussing these generalization measures , it should be made clear how each of the information matrices are defined . In this paper , uncentered gradient covariance matrix is denoted as C ( θ ) . We define qθ as a model distribution . Furthermore , we employ the data distributions p̂ and p introduced in the previous section as the empirical and true data distributions respectively . Matrices H ( θ ) , C ( θ ) and F ( θ ) are then defined as : H ( θ ) = Ep [ ∂2l ( y , f ( x , θ ) ) ∂θ∂θT ] ∈ Rd×d , C ( θ ) = Ep [ ∂l ( y , f ( x , θ ) ) ∂θ ∂l ( y , f ( x , θ ) ) ∂θT ] ∈ Rd×d , F ( θ ) = Eqθ [ ∂l ( y , f ( x , θ ) ) ∂θ ∂l ( y , f ( x , θ ) ) ∂θT ] ∈ Rd×d ( 2 ) The conditions under which these matrices are equal will be discussed in detail in section 3.1 . The relation betweenC ( θ ) andF ( θ ) is often misunderstood because they both involve the outer product of the gradients but they a different distribution when computing the expectation . As a subsequent study , Novak et al . ( 2018a ) concluded that consideration of either H ( θ ) or C ( θ ) alone is insufficient to estimate the generalization of DNNs and that both are essential . In particular , H ( θ ) is a value that does not depend on the distribution of input data ; however , as the generalization ability depends on the distribution of the data , it is also natural to considerC ( θ ) , which is related to noise in the gradient . Furthermore , as supporting evidence of Novak et al . ( 2018a ) ’ s claim , Thomas et al . ( 2020 ) showed empirically the effectiveness of TIC , a generalization measure that considers both H ( θ ) and C ( θ ) expressed by the equation 1 . However , Thomas et al . ( 2020 ) ’ s work only experimented with very small-scale NNs because it is challenging to calculate TIC with DNNs of practical size . As a matter of fact , even the ResNet-8 model used in the small-scale image classification benchmark CIFAR10 is not feasible , as it requires nearly 200TB of memory to calculate the TIC exactly . Remark 2.1 . It should also be noted that TIC is an information criterion for regular models , not for singular models such as DNNs , and its theoretical justification in the domain of DNNs is not clear . One of the characteristics of singular models is that the Fisher information matrix is not positive definite .

Summary: This paper focuses on a fascinating, challenging problem: evaluating the generalization performances of neural networks based on the training set with some criteria. Specifically, the authors propose to measure the generalization ability via Takeuchi’s Information Criteria (TIC). Besides, the authors approximate the TIC to accelerate the process of estimation. The proposed method is promising and works well when d/n is large (where d = #parameters and n = #samples).  However, I am still puzzled about some details. I will raise my score if the authors could clarify those concerns (see the following).