Year: 2021

Abstract:
Generalization measures are intensively studied in the machine learning community for better modeling generalization gaps. However, establishing a reliable generalization measure for statistical singular models such as deep neural networks (DNNs) is challenging due to the complex nature of the singular models. We focus on a classical measure called Takeuchi’s Information Criteria (TIC) to investigate allowed conditions in which the criteria can well explain generalization gaps caused by DNNs. In fact, the theory indicates the applicability of TIC near the neural tangent kernel (NTK) regime. Experimentally, we trained more than 5,000 DNN models with 12 DNN architectures including large models (e.g., VGG16) and 4 datasets, and estimated corresponding TICs in order to comprehensively study the relationship between the generalization gap and the TIC estimates. We examine several approximation methods to estimate TIC with feasible computational load and investigate the accuracy trade-off. Experimental results indicate that estimated TIC well correlates generalization gaps under the conditions that are close to NTK regime. Outside the NTK regime, such correlation disappears, shown theoretically and empirically. We further demonstrate that TIC can yield better trial pruning ability for hyperparameter optimization over existing methods.

heading: 1 INTRODUCTION
text: Deep neural networks (DNNs) have been exhibiting great generalization abilities in many applications, but the mechanism of the generalization has not been fully understood yet (Neyshabur et al., 2014; Zhang et al., 2016; Recht et al., 2019). Establishing a reliable generalization measure is an important research topic for generating a good model from limited data resources, including an application of hyperparameter search. Many attempts (Arora et al., 2018; Wei & Ma, 2019; Neyshabur et al., 2018) have been taken to better understand the generalization phenomenon in deep learning models from theoretical points of view. From empirical points of view, there have been intensive studies (Keskar et al., 2016; Liang et al., 2019; Bartlett et al., 2017) in search of learning conditions that likely yield high model performance.
Work by (Jiang et al., 2019) indicated that a measure that includes both hessian H(θ)1 and covariance C(θ) defined from the loss and the network parameters θ near a local minimum may potentially show good correlation with generalization performance. Another study indicated that use of only a single measure, either HessianH(θ) or covariance C(θ), fails to capture the generalization performance (Novak et al., 2018a).
The generalization gap inherently stems from a discrepancy between the empirical and the true data distribution. A minimizer of the empirical loss will be affected by the noise due to a finite number of samples and by the form of the loss landscape near the minimum. The former can be characterized as noise (C(θ)) and the latter as curvature (H(θ)).
Taking these findings into account, we sought to model the generalization gap, then found that a classical information criterion called Takeuchi’s Information Criteria (TIC) (Takeuchi, 1976) ex-
1H(θ) andC(θ) are defined in equation 2
presses generalization gap in the neural tangent kernel (NTK) regime. TIC has the following form
TIC(θ)︸ ︷︷ ︸ Information Criteria = −Ep̂[l(y, f(x,θ))]︸ ︷︷ ︸ Mean Empirical Error
+ Tr ( H(θ)−1C(θ) )︸ ︷︷ ︸ Estimated Bias Term , (1)
where f is a smooth function over θ ∈ Rd with input x and target y, and l is the negative loglikelihood, also denoted as loss function. The first term on the right-hand side is the log-likelihood, which takes the expectation over an empirical data distribution (xi, yi) ∼ p̂. For the later discussion, we use θ̂ minimizes the empirical loss; i.e., θ̂ = arg minθEp̂[l(y, f(x,θ))], and θ
∗ as the parameters that maximizes the likelihood with respect to the true data distribution (x, y) ∼ p; i.e., θ∗ = arg minθEp[l(y, f(x,θ))].
For a DNN of practical size, exact computations of the matrices H(θ), C(θ) are nearly infeasible due to large dimensionality. To make the computation feasible, we adopted a strategy of using shared components of the matrix to estimate TIC with fewer computations, based on a relationship among matrices such as H(θ), C(θ), and Fisher Information Matrix F (θ). To further reduce the computational costs for the bias term, we examined methods using approximations and lower bounds so that TIC estimations for DNNs of practical sizes are feasible.
In this work, we make the following contributions:
• We provide empirical and theoretical evidence that TIC is highly correlated with generalization gap of DNNs that are close to NTK regime, despite the fact that TIC is not originally designed for singular model such as general DNNs.
• We conduct comprehensive experiments in which more than 5,000 models, including ones close to NTK regime, with totally 12 DNN architectures, 4 datasets and 15 training settings are trained, and corresponding TICs are estimated with approximation techniques, to clarify conditions that TIC can well explain generalization gaps.
• We use TIC as a threshold for pruning poorly performing trial models during hyperparameter optimization (HPO) and show that it can successfully prevent promising candidates from being pruned prematurely.

heading: 2 GENERALIZATION MEASURES
text: Generalization measures measure the generalization ability of statistical models. Typically, the generalization gap, which is defined as the difference between training loss and validation loss, is used to quantify the generalization ability.

heading: 2.1 WHICH GENERALIZATION MEASURE IS PROSPECTIVE?
text: To answer this question, before demonstrating the effectiveness of TIC, we highlight the development of research in this area and the motivation behind this work. For understanding generalization behavior, there are two major approaches by quantifying generalization bounds and complexity measures.
Approach of quantifying generalization bounds is pursued by theoretical groups to prove the bound of the generalization gap (Dziugaite & Roy, 2017). Although tight bounds can be proven, they are often based on assumptions that do not apply to practical DNN settings. In addition, no bounds have been shown to describe the performance of the current DNNs to a satisfactory level.
On the other hand, approach of quantifying complexity measures, which do not necessarily certify bounds, follows the principle of Occam’s razor and evaluates the complexity of the model. Theoretically motivated complexity measures, including VC-dimension (Vapnik & Chervonenkis, 2015), PAC-Bayesian framework (McAllester, 1999), the norm of parameters (Neyshabur et al., 2015), are often discussed as significant components of generalization bounds, and a monotonic relationship between complexity measures and generalizations is mathematically established. In contrast, empirically motivated generalization measures, such as sharpness (Keskar et al., 2016), are justified by experiments and observations. In particular, for DNNs, Jiang et al. (2019) have conducted exhaustive experiments to evaluate the effectiveness of generalization measures for three groups: norm-based measure, sharpness-based measure, and noise-based measure.
• Norm-based measure: |θ|. Most of the proposed norm-based measures are based on the FisherRao Metric (Liang et al., 2019), which does not capture generalization well. In particular, it has been reported that spectral complexity such as product of spectral norms of the layers (Bartlett et al., 2017) has a strong negative correlation with generalization. It is impossible to explain the success of DNN models with huge parameter sizes in recent years with these metrics.
• Sharpness-based measure: H(θ). Sharpness-based metrics, such as sharp minima and flat minima (Keskar et al., 2016) and PAC-Bayesian framework (McAllester, 1999), are not only associated with intuitive understanding but also empirically show a strong correlation with the generalization gap. However, some model architectures are known to show poor correlation (Dinh et al., 2017).
• Noise-based measure: C(θ). Experimental results show that generalization measure based on gradient has potential (Jiang et al., 2019). In particular, in their experiments, they observe that the variance of the gradient captures Sharpness, but they suggest that this is not a good generalization measure depending on the architecture of the model.
These results suggest that studying generalization measures that can be estimated using H(θ) and C(θ) is prospective. However, since the combination ofH(θ) andC(θ) seen in TIC is not feasible to compute for practical DNN settings, so it was outside the scope of (Jiang et al., 2019).

heading: 2.2 INFORMATION MATRIX: ELEMENTS OF GENERALIZATION MEASURES
text: Previous research has highlighted information matrices such as H(θ) and C(θ) in generalization measures in DNNs. Thomas et al. (2020); Kunstner et al. (2019) remarked that these matrices are often confused and misused, for example in the field of optimization, leading to wrong conclusions, even though these matrices play an essential role in the study of DNNs, especially in optimization (Amari et al., 2020; Martens & Grosse, 2015a), understanding implicit regularization in SGD (Wen et al., 2019; Zhu et al., 2019), and Bayesian inference (Zhang et al., 2018). Before discussing these generalization measures, it should be made clear how each of the information matrices are defined.
In this paper, uncentered gradient covariance matrix is denoted as C(θ). We define qθ as a model distribution. Furthermore, we employ the data distributions p̂ and p introduced in the previous section as the empirical and true data distributions respectively. Matrices H(θ), C(θ) and F (θ) are then defined as:
H(θ) = Ep [ ∂2l(y, f(x,θ))
∂θ∂θT
] ∈ Rd×d,
C(θ) = Ep [ ∂l(y, f(x,θ))
∂θ
∂l(y, f(x,θ))
∂θT
] ∈ Rd×d,
F (θ) = Eqθ [ ∂l(y, f(x,θ))
∂θ
∂l(y, f(x,θ))
∂θT
] ∈ Rd×d
(2)
The conditions under which these matrices are equal will be discussed in detail in section 3.1. The relation betweenC(θ) andF (θ) is often misunderstood because they both involve the outer product of the gradients but they a different distribution when computing the expectation.
As a subsequent study, Novak et al. (2018a) concluded that consideration of either H(θ) or C(θ) alone is insufficient to estimate the generalization of DNNs and that both are essential. In particular, H(θ) is a value that does not depend on the distribution of input data; however, as the generalization ability depends on the distribution of the data, it is also natural to considerC(θ), which is related to noise in the gradient. Furthermore, as supporting evidence of Novak et al. (2018a)’s claim, Thomas et al. (2020) showed empirically the effectiveness of TIC, a generalization measure that considers both H(θ) and C(θ) expressed by the equation 1. However, Thomas et al. (2020)’s work only experimented with very small-scale NNs because it is challenging to calculate TIC with DNNs of practical size. As a matter of fact, even the ResNet-8 model used in the small-scale image classification benchmark CIFAR10 is not feasible, as it requires nearly 200TB of memory to calculate the TIC exactly. Remark 2.1. It should also be noted that TIC is an information criterion for regular models, not for singular models such as DNNs, and its theoretical justification in the domain of DNNs is not clear. One of the characteristics of singular models is that the Fisher information matrix is not positive definite.

heading: 2.3 TIC IS DERIVED AS GENERALIZATION GAP IN NTK REGIME
text: This section outlines the derivation of the definition of TIC in equation 1, considering the generalization gap of DNNs in the framework of NTK’s regime. We employ the setting introduced in section 1, f is a smooth function over θ ∈ Rd, a parameter of the statistical model. First, we further assume that the following holds for f and θ ∈ Rd in the NTK regime. Assumption 2.1.
(A1) Global convergence: the model has only one possible solution. However, it is not required to be qθ = p (allowing for misspecified situation).
(A2) Asymptotic normality: the maximum likelihood estimator θ̂ from the empirical data distribution p̂ and the maximum likelihood estimator θ∗ in the true data distribution p satisfy asymptotic normality.
Proposition 2.1 (Generalization Gap in NTK Regime is equal to TIC).
Under the assumptions (A1) and (A2), the estimated bias b (i.e. generalization gap) when evaluating using empirical data distribution p̂ would then be as follows. b = Ep [ Ep̂[l(y, f(x, θ̂))]− Ep[l(y, f(x, θ̂))] ] = Tr ( Hp(θ ∗)−1Cp(θ ∗) ) (3)
Where Hp(θ∗) and Cp(θ∗) are the Hessian and Covariance respectively with regards to θ∗ over true data distribution p. However, as the true data distribution p and parameter θ∗ which maximizes the likelihood for that data distribution are unknown, the expected value in the empirical data distribution p̂ and parameter θ̂ are generally used as a consistent estimator, which is consistent with the TIC. A more detailed proof is given in Appendix A.1.2. Remark 2.2. The bias term of TIC is formulated as Tr ( H(θ)−1C(θ) ) . However, since there is
no guarantee that H(θ) is positive definite in practice. To prevent this problem, the addition of a small identity matrix, called damping, is performed as H̃(θ)−1 = (H(θ) + λI)−1. Alternatively, consider the case where the TIC is calculated by approximation with a matrix of only the diagonal components of the respective matrices, as Tr ( H(θ)−1C(θ) ) ≈ Tr ( Hdiag(θ) −1Cdiag(θ) )
In this case, the following lower bound is given for the diagonal approximated TIC.
Tr ( Hdiag(θ) −1Cdiag(θ) ) > Tr(Cdiag(θ))
Tr(Hdiag(θ)) =
Tr(C(θ)) Tr(H(θ)) (4)
Remark 2.3. We note that not all DNNs are in the NTK regime. One indicator of whether a DNN is in the NTK regime is the ratio of the number of model parameters to the number of data. In general, unconstrained DNNs are singular models, so WAIC (Watanabe, 2013) is appropriate instead of TIC or AIC (Akaike, 1998), but computational cost of WAIC is way too high to perform a wide range of learning experiments. Furthermore, when the loss function includes a regularization term, GIC (Konishi & Kitagawa, 1996) is technically appropriate instead of TIC, but it has a disadvantage that the calculation is further complicated.

heading: 3 APPROXIMATION OF TIC

heading: 3.1 HESSIAN, GENERALIZED GAUSS-NEWTON MATRIX (GGN) AND FIM
text: In this section, we describe the conditions for which the Hessian, GGN, and FIM become equivalent. This equivalence can be exploited to reduce the computational cost of computing the TIC. TIC requires the computation of H(θ) and C(θ), but the computational cost of H(θ) is relatively high. For NNs that consist of linear, convolutional, and pooling layers, along with piecewise linear activations, the Hessian is equal to the GGN (Schraudolph, 2002). This actually holds true for most CNNs used in practice. The GGN is an extension of the Gauss-Newton matrix G̃(θ) = Ep [ (Jθ) TJθ ] .
G(θ) = Ep [ (Jθ) THfJθ ]
(5)
Where Hf is the Hessian of l(y, f(x,θ)) and Jθ is Jacobian of f(x,θ) with respect to θ. Furthermore, the GGN is equal to the FIM for any NN that uses the softnax cross entropy. Therefore, we can assume the following for most practical DNN problem settings.
Assumption 3.1.
(B1) Loss function: l is the softmax cross-entropy function
(B2) Activation function: inside f , all activation functions’ second derivative are always zero, such as ReLU or the identity function.
Proposition 3.1 (H(θ) is equal to F (θ) through GGN).
Under the assumption of (B1) and (B2),H(θ) and F (θ) are exactly equal through GGN. They are also guaranteed to be positive semi-definite.
H(θ) = G(θ) = F (θ) (6)
A more detailed proof is given in Martens (2020).

heading: 3.2 APPROXIMATION OF MATRICES AND TRACE ESTIMATION
text: As noted in section 2.2, information matrices are in demand for many applications, including TIC. However, for a model with a large number of parameters d such as a DNN, it is necessary to compute a matrix with size of d2. For this reason, approximation methods ranging from approximating the information matrix itself (Le Roux et al., 2007) to approximating the product of the information matrix, and the vector directly is used in optimization (Pearlmutter, 1994) and other applications. We propose the following approximation method to calculate the TIC and experimentally verify the trade-off between accuracy and computation time.
• Replacing H(θ) in F (θ) and fast estimation of F (θ) in Monte-Carlo sampling. As shown in equation 6, F (θ) can be used in place of H(θ) under the (B1) and (B2) assumption. We use this property to speed up the calculation by simultaneously computing C(θ) and F (θ), which have a common term. Furthermore, since the number of classes for the classification task is 10 in MNIST and 100 in CIFAR100, the computational cost of F (θ) is huge, so we approximate F (θ) using Fmmc(θ), which is a Monte Carlo approximation. Martens & Grosse (2015b) use m = 1 in the practical setting. We follow this setting F1mc(θ) for the approximation of F (θ).
• Block-diagonalization and diagonalization. In NTK Regime, the correlation between layers is ignored, so block-diagonalization is a reasonable approximation method. The computational complexity can be reduced from O(d3) to O(d3l )
2 by the block-diagonal approximation. Diagonalization is a simple approximation; it ignores the correlation between DNN units. It has been reported to be sufficient for some applications (Singh & Alistarh, 2020). It can also be calculated as a sum-of-products operation on vectors rather than matrices, significantly reducing computational complexity and memory consumption. In particular, by using the diagonal approximation, the inverse calculation ofH(θ) can be reduced from O(d3) to the order of O(d).
• Lower bound of diagonalization. As shown in equation 4, by giving the lower bound of the diagonal approximation, it is possible to calculate the trace of each matrix by calculating and dividing the trace of each matrix without calculating the diagonal component of the matrix. In other words, it is possible to calculate without considering whether F (θ) is positive definite.
• Hutchinson’s method for estimating Tr(H(θ)) in fast. Furthermore, rather than approximating the matrix itself, we will introduce a method to accelerate the computation of its eigenvalues and trace. For optimization in deep learning, it is enough to calculate not the Hessian itself but the product of the Hessian and an arbitrary vector (Hessian vector product; Hvp). In order to compute Hvp exactly, Pearlmutter (1994) proposed a fast algorithm to compute Hvp in NNs during backpropagation. This Hvp can be applied to non-optimization applications, such as approximating Tr(H(θ)) (Avron & Toledo, 2011). Hutchinson’s method (Hutchinson, 1989) approximates the expectation value of the quadratic form of the Hessian and the Rademacher random vector (each element takes 1 or −1 with probability 1/2).
2l is the number of parameters for the layer with the largest number of parameters in the network.

heading: 4 EXPERIMENTS

heading: 4.1 OVERVIEW
text: The goal of our paper is to elucidate the correlation between the TIC estimates and the generalization gap. To make our study of TIC as comprehensive as possible, we trained on 4 different data sets and 12 different DNN models. Using these combinations, we searched for hyperparameters for each of the 15 problem settings and evaluated the parameters of the trained models. By comparing these results, we can observe how the effectiveness of TIC changes with the model and problem settings. In our experiments, the bias term of TIC is estimated by using validation data, and the generalization gap is the absolute value of the difference in loss between training and test data, using all of the data in each dataset, not just a part of the data. The problem settings for the experiment can be divided into two main categories along with dataset and model size as table 1.
In particular, ResNet-8, which is commonly used as a benchmark for training CIFAR10, requires over 200TB of memory to compute exact H(θ). Even state-of-the-art GPU NVIDIA A100 is impractical since it has only 80 GB of device memory. Hence, as small-scale experiment, we use a small dataset called TinyMNIST to limit the size of the DNN model, which is a resized version of the MNIST image, which reduces the dimension of the input layer of DNN, to compare our approximation method and exact calculation. As practical-scale experiments, we evaluated the real-world datasets and DNN models. We used diagonal approximations and their lower bound approximations to estimate TIC.

heading: 4.2 SMALL SCALE EXPERIMENTS: COMPARING APPROXIMATION AND EXACT
text: As small-scale experiments, we trained Tiny MNIST on 5 experimental settings: 3-LNNs and NNs, each w/ and w/o SC, and a wide model, 2-NNs without SC. Afterwards, we evaluated the approximation of Tr ( H(θ)−1C(θ) ) , the bias term of TIC, for H(θ) and C(θ), using block-diagonal
42-NN and 3-LNN denote 2-Layer Non Linear Neural Network and 3-Layer Linear Neural Network respectively. SC denotes Skip-Connection
4BN denotes Batch Normalization.

heading: 100 101
text: approximation, diagonal approximation, and its lower bound. Furthermore, as mentioned in equation 6, for the purpose of speeding up the process, we also estimate TIC using F (θ), which shares the same elements to be calculated as C(θ) as an alternative.
Remark 4.1. It should be noted that the above five settings are different from the situation of NTK, since d p. However, we observed that the estimation of TIC was effective for LNNs.
First, we show the results of our experiments on the quality of the approximations. In general, from the exact computation to the block-diagonal approximation, i.e., the approximation which ignores the correlation between layers, we can confirm that the value and the rank correlation are kept. As for the LNN, the rank correlation is maintained for the block-diagonal approximation, the diagonal approximation, and its lower bound, though the value fluctuates. On the other hand, in the case of NN w/ SC, we confirmed that the rank correlation is maintained between exact and block-diagonal approximation, between diagonal approximation and its lower bound. These results show that LNN or NN with more layers and SC has a trend of the higher approximation quality.
Then we explain the correlation between the TIC estimates and the generalization gap. We observed that LNN is in the effective regime of TIC and has a high correlation with the generalization gap in all approximations. For NNs, similarly high correlations were observed for the models w/ SC. For the 3-NN w/o SC, the results were such that the inverse correlation was observed even in the exact case. In the case of 2-NN, the approximate correlation was also collapsed, resulting in no correlation with the generalization gap.
From these results, we conclude that the performance of TIC on the correlation with the generalization gap is higher for NN models with more layers and SC, and the correlation does not change significantly before and after the approximation.

heading: 4.3 PRACTICAL SCALE EXPERIMENTS: CORRELATION TO GENERALIZATION GAP AND TIC LOWER BOUND, TIC WITH DIAGONAL APPROXIMATION
text: As practical-scale experiments, we experimented with the problems where d p, which is considered to be NTK’s Regime. Contrary to small-scale experiments, we used MNIST, CIFAR10, and CIFAR100 datasets to evaluate practical settings.
First, we show the case of MNIST. The settings of LNN show a strong correlation with the generalization gap in the lower bound approximation as well as in the small-scale experiment. In the case of the NN model, a strong correlation with the generalization gap is observed, unlike in the small-scale setting. Furthermore, in the case of NN and LNN w/ SC, it has less variance and shows a stronger correlation with the generalization gap. In the Simple CNN case, the correlation with the generalization gap is weaker than in previous cases but still shows a correlation. Also, there is no correlation with the generalization gap in the case of the value of Tr (H(θ)), Tr (F (θ)), Tr (C(θ)) itself respectively. Detailed experimental results are shown in figure 13 in the Appendix D.3.
In the cases of CIFAR10 and CIFAR100, both the measures using lower bound and the diagonal approximation show a high correlation with the generalization gap. For LNNs, the correlation is more linear in the case w/ SC. For VGG16 and ResNet8, the correlation is not as good as for LNN, but we confirmed the effectiveness of TIC in NTK’s regime. Furthermore, no correlation was found between the generalization gap and trace itself, respectively. These trace values have different patterns depending on the network, and it was found that this single factor alone is insufficient for estimating the generalization gap.
Remark 4.2. It should be noted that TIC estimates captured the trend of the generalization gap in the training process as shown in figure 4.

heading: 4.4 CALCULATION RUNTIME MEASUMENT EXPERIMENTS
text: Our runtime measurement experiments were run on an NVIDIA Tesla V100 16GB GPUs, with an average of 10 trials each. Significant speedup was achieved by approximating the shape of the matrix, replacing H(θ) by F (θ), and Monte Carlo estimation of F (θ), as shown in 3. Even in the case of a small-scale problem setting, the diagonal approximation with F (θ) and C(θ) is 50 times faster than the exact version, while maintaining the rank correlation with H(θ) and C(θ). However, since the number of parameters in the small problem setting is at most 720, and VGG16 has 186,530 times as many parameters, the effect of increasing the computational order from O(d3) toO(d) is more significant in the large-scale problem setting. The full details are shown in Appendix D.4. Notably, this speedup by using F (θ) and C(θ) as a set instead of H(θ) and C(θ), and the method of approximating the matrix form to drop the computation order dramatically reduces the computation time.

heading: 5 APPLICATION TO HYPERPARAMETER OPTIMIZATION
text: In previous sections, we have demonstrated that TIC is a reasonable estimator of the generalization gap that is also effective in the training process and can be computed fast. Motivated by these, in this section, we employ the TIC values on the training processes to accelerate hyperparameter optimization (HPO). HPO is an essential task to achieve good performance in a wide range of machine learning algorithms (Feurer & Hutter, 2019). In particular, the performance of DNNs depends significantly on the selection of the hyperparameters, such as learning rates, weight decay, and momentum (Lucic et al., 2018; Henderson et al., 2018; Dacrema et al., 2019).
The Successive halving algorithm (SHA) (Jamieson & Talwalkar, 2016) shows promising performance in HPO by utilizing the iterative structure of DNNs. SHA prunes unpromising hyperparameters at early stage by utilizing not only a final loss but also losses in training process. The validation loss obtained by the hold out method is usually used as the intermediate loss for SHA. However, the validation loss is often numerically unstable, as shown in Figure 5.
To achieve stable optimization in SHA, we apply the TIC values for the intermediate loss. One advantage of using TIC is that it can take into account the variance as bias term in equation 1, which was not taken into account in the validation loss obtained by the hold out method. In particular, TIC is known to be asymptotically close to leave-one-out cross-validation (LOOCV) (Stone, 1977), and is superior to Hold-Out in terms of the order of estimates error. Details are given in Appendix B. We conduct an experiment to investigate the effectiveness of using the TIC values in SHA. Figure 5 shows the result of the experiment. The TIC values with the proposed approximation method can select 1st top trial, while the traditional method (SHA + the validation loss obtained by the hold out method) selects 3rd top trial.

heading: 6 CONCLUSION AND DISCUSSION
text: This study conducted a comprehensive experiment and observed that the TIC approximation method captures the generalization gap, even in the practical DNN setting close to the NTK regime. We have shown that the generalization gap could be captured in the training process, even if the model is not completed to train. Based on these results, we tested the validity of using TIC as an assessment value for HPO branch pruning and confirmed a valid case. It is challenging to establish a theory to discuss in the active regime (outside of the NTK regime) for future work. Especially, WAIC can handle singular models and applied to DNN; the difficulty arises for calculation; thus, it is required to have an approximation method. Still, it is necessary to bridge the theory of DNN for the validity of the approximation.

heading: APPENDIX

heading: A PROOFS

heading: A.1 DERIVATION OF THE TIC IN NTK REGIME

heading: A.1.1 NTK: NEURAL TANGENT KERNEL
text: In general, DNNs have a large number of parameters p, compared to the number of data, n, causing them to memorize data and hurt their generalization ability. Furthermore, all data are subjected to nonlinear transformations, which results in the problem of minimizing a nonconvex objective function. Due to this difficulty in analyzing the training dynamics of DNNs, the reasons for the generalization of practical DNNs and the mysteries related to the guarantee of global convergence remain unsolved. However, a theoretical framework called Neural Tangent Kernel (NTK) (Jacot et al., 2018) has been developed to analyze the training dynamics of gradient descent in DNNs with a sufficiently large width.
Assuming that the loss function to be minimized in NN training is L = 12‖f(x;θ) − y‖ 2, and the parameters are updated by the gradient method with the learning rate η, the amount of change is ∆θ,
∆θ = −η ∂L(θ) ∂θ = −η∇θL(θ) (7)
Continuing this equation using the continuous time (training step of NN) t, we obtain
∂θ ∂t = −η∇θL(θ(t)) (8)
= −ηJt(x;θ)T∇ft(x)L(ft(x)) (9)
where Jt(x;θ) is ∂f(x;θ) ∂θ = ∇θft(x).
Second, considering the time variation of the output of the function instead of the parameters we get
∂ft(x;θ)
∂t =
∂ft(x;θ)
∂θ
∂θ ∂t (10)
= −ηJt(x;θ)Jt(x;θ)T∇ft(x)L(ft(x)) (11) = −ηKt(x, x)∇ft(x)L(ft(x)) (12)
where Kt(x, x) is Jt(x;θ)Jt(x;θ)T = ∑num of layer layer=1 Jt (x;θlayer) Jt (x;θlayer) T
The problem here is that Kt(x, x) which is the NTK at time t depends on θ and x. However, it was shown that if the width of the randomly initialized NN is large enough, it will be Kt(x, x) ∼ K0(x, x) (Jacot et al., 2018). Given the output of the neural network with its first-order Taylor expansion, we have
f lint (x;θt) ≈ f0 (x;θ0) +∇θf0 (x;θ0) T (θt − θ0) (13)
The dynamics of training can be expressed in the following way. θt = θ0 −∇θf0(x)TK−10 ( I − e−ηK0t ) (f0(x)− y) (14)
f lint (x) = ( I − e−ηK0t ) y + e−ηK0tf0(x) (15)
Therefore, NTK theory determines the dynamics of gradients in function space by introducing the NTK regime, which allows us to assume that the weights follow a Gaussian process even in training progresses, based on the theory that the random initialized NN can be considered as a Gaussian process when the hidden layers become infinite in the study of Lee et al. (2018); Novak et al. (2018b).
NTK allows us to prove the global convergence of gradient descent, and furthermore, the equivalence between the trained model and the Gaussian process can be used to explain the generalization performance of DNN. Besides, NTK was extended to CNNs (Arora et al., 2019) and RNNs (Yang, 2019) as well as MLPs, and exhaustive experiments (Lee et al., 2019) were conducted.

heading: A.1.2 PRELIMINARIES FOR TIC IN NTK REGIME
text: TIC requires that the statistical model be a regular model. However, DNNs are generally singular models. The requirements for a regular model are as follows,
• The posterior distribution of the parameters can be approximated by a Gaussian distribution, and the number of samples is sufficiently large (as n increases, the prior distribution is ignored).
• There is only one optimal solution θ̂ for arg max `(θ). • Hp(θ∗) is positive definite.
In machine learning, we often pursue to minimize the negative log-likelihood using it as a loss function. Let f be the predictive distribution of the model over the p-dimensional parameters θ ∈ Θ ⊂ Rp and the true distribution g . We can compare the models by measuring the KL divergence of f and g to see how well f approximates the true distribution g as a predictive distribution.
DKL(g, f) = Ep [ log
g(y|x) f(y|x,θ)
] (16)
= Ep[log g(y|x)]− Ep[log f(y|x,θ)] (17)
Since the first term of equation 17 is independent of θ, the model is better if it maximizes the second term, i.e., the higher the mean log-likelihood L(θ) = Ep[log f(y|x,θ)]. The mean log-likelihood L(θ) is also an unknown quantity that cannot be calculated as it depends on the true distribution of the data p . However, suppose it is possible to obtain a valid estimator of the mean log-likelihood using the empirical distribution of the data p̂.In that case, it can be used as a criterion for evaluating the model.
In model selection, we assume an output of DNN model f(y|x,θ), which is estimated by the maximum likelihood method. We can calculate f(y|x, θ̂) by replacing the unknown parameters θ in the probability distribution with the maximum likelihood estimator θ̂ .
θ̂ := arg max θ∈Θ `(θ) (18)
Where `(θ)=Ep̂[log f(y|x,θ)]=1n ∑n i=1 log f(yi|xi,θ) is the likelihood function over θ ∈ Θ.
Let Sn = {(x1, y1), (x2, y2), . . . (xn, yn)} be the data observed according to the true data distribution p. Let p̂ be the empirical distribution based on this Sn. By the law of large numbers, the mean of Ep̂[log f(y|x, θ̂)] = 1n ∑n i=1 log f(yi|xi, θ̂) converges in probability to its expected value Ep[log f(y|x, θ̂)] when the number of data n becomes infinitely large.
1
n n∑ i=1 log f(yi|xi, θ̂) −−−−−→ n→+∞ Ep[log f(y|x, θ̂)] (19)
Therefore, the estimator based on the empirical distribution function in equation 19 is a natural estimator of the mean log-likelihood. While Ep̂[log f(y|x, θ̂)] is a natural estimator of Ep[log f(y|x, θ̂)], θ̂ is a model parameter estimated using empirical data (xi, yi) ∼ p̂. The data is also used to evaluate the mean log-likelihood of f(y|x, θ̂) of the model in terms of prediction, although a fair model selection is not possible.
Therefore, it is necessary to evaluate and correct for this bias for fair model selection. The bias of estimating the mean log-likelihood Ep[log f(y|x, θ̂)] with equation Ep̂[log f(y|x, θ̂)] is formulated as follows:
b=nEp [ Ep̂[log f(y|x, θ̂)]− Ep[log f(y|x, θ̂)] ] (20)
=nEp [ `(θ̂)− L(θ̂) ] (21)
=nEp [ `(θ̂)− `(θ∗) ] (22)
+nEp [`(θ∗)− L(θ∗)] (23) +nEp [ L(θ∗)− L(θ̂) ] (24)
Where θ∗ is the maximum likelihood estimator of L(θ), thus, equation 21 can be decomposed as equation 22, 23, 24. Also, equation 23 converges to 0 ,since taking the expectation of the left-hand side by p coincides with the right-hand side, and equations 22 and 24 converge to 1 2Tr ( Hp(θ ∗)−1Dp(θ ∗) )
in n→∞, respectively. Technically, this asymptotic validity holds under the regularity condition (White, 1982).

heading: A.1.3 APPLY NTK REGIME FOR TIC DERIVATION
text: Now we satisfy the above condition by using the training dynamics of DNN in the NTK regime. Specifically, the regime in NTK satisfies the first condition because it uses a locally linear approximation, as in equation 13, and considers the training of the NN as a Gaussian process. Also, as shown in equation 14, the optimal solution at the t-th step is uniquely determined, and the optimization is convex.
The positive definiteness of Kt(x, x) is proved in Appendix A4 of Jacot et al. (2018), under the assumption of non-polynomial Lipschitz nonlinearity. From the definition ofKt(x, x), the eigenvalues of Fisher Information Matrix (FIM) are positive definite in NTK Regime because Kt(x, x) and FIM have a duality that shares eigenvalues.
This condition is true when the DNN is considered to be in the NTK Regime, i.e., when Assumption 2.1 is satisfied.
Figure 6 shows a schematic diagram of the bias term b. The matrices Hp(θ∗) and Dp(θ∗) is as follows:
Dp(θ ∗) = Ep
[ ∂ log f(y|x,θ)
∂θ ∂ log f(y|x,θ) ∂θT ∣∣∣ θ=θ∗ ] Hp(θ ∗) = Ep [ ∂2 log f(y|x,θ)
∂θ∂θT
∣∣∣ θ=θ∗ ] (25) If the true distribution g is included in the assumed statistical model f(y|x,θ), then Dp(θ∗) = Hp(θ
∗) is valid and b = Tr(I) = d, and thus AIC can be derived. In the DNN setting, this assumption does not hold, i.e., we need to use the respective matrices for the misspecified situation.
Since the bias b depends on the true data distribution p, it needs to be estimated based on the observed data. Assuming that the consistent estimators forDp(θ∗) andHp(θ∗) are C(θ̂) andH(θ̂), respectively, the estimates in equation 20 are as follows:
C(θ̂) = 1n ∑n i=1 ∂ log f(yi|xi,θ) ∂θ ∂ log f(yi|xi,θ) ∂θT ∣∣∣ θ=θ̂
H(θ̂) = 1n ∑n i=1 ∂2 log f(yi|xi,θ) ∂θ∂θT ∣∣∣ θ=θ̂
(26)
Using equation 2, b̂ as an estimate of bias b can be described as follows.
b̂ = Tr ( H(θ̂)−1C(θ̂) ) (27)
The term of b is called Moody’s effective number of parameters (Moody, 1992). The TIC is derived in equation 1 by estimating the asymptotic bias of the mean log-likelihood with the log-likelihood of the statistical model.

heading: B ASYMPTOTIC EQUIVALENCE OF TIC TO CROSS-VALIDATION
text: Since deep learning usually requires a lot of data and a huge amount of time for training, the HoldOut method is commonly used to divide the training data into train data for training, validation data for model selection, especially for hyperparameter optimization, and test data to verify the performance of the model. This method is relatively fast, but its evaluation varies depending on how the data is divided, and it is not used when the number of data is small. In K-fold cross-validation, the entire train data is divided into K pieces. Then, one of them is used as the validation data, and the remaining K-1 groups are decomposed into training data. The validation data and training data are swapped and repeated, and the verification is repeated so that all cases become validation cases. The leave-one-out cross-validation (LOOCV) is a method that uses only one piece of the entire train data as validation data. LOOCV is empirically known to have high performance and is often used when the overall data is small. If the number of data is n, the bias of the estimation error is O(1/ √ n) for the Hold-Out method and O(1/n) for LOOCV (Stone, 1977). However, LOOCV requires n times the computational cost. In a case such as ImageNet-1K, 1.2 million images can be used as training data, and the current trend is to use the Hold-Out method, which allows the estimation error to be O(1/ √ n) with small computational cost, instead of reducing the estimation error to O(1/n) at the expense of huge computational cost.

heading: C TIC EXPERIMENTAL DETAILS
text: C.1 IMPLEMENTATION AND ENVIRONMENT FOR EXPERIMENT
We perform our experiment with supercomputer A (These names will be deanonymized after publication). For supercomputer A, each node is composed of NVIDIA Tesla V100×4GPU and Intel Xeon Gold 6148 2.4 GHz, 20 Cores×2CPU. As a software environment, we use Red Hat 4.8.5, gcc 7.4, Python 3.6.5, Pytorch 1.6.0, cuDNN 7.6.2, and CUDA 10.0. Our code can be found at the link below. https://anonymous.4open.science/r/TIC-in-NTKRegime-02A2/

heading: C.2 HYPERPARAMETERS AND DETAILED CONFIGURATION
text: We will report the hyperparameter’s search space. We search learning rate η, learning rate decay rate ρ and the timing to decay learning rate δ, and regularization coefficient of weight decay λ. When δ = 0.7, it means that the learning rate decays when training passes 70 % of the total iterations. Furthermore, a parameter to control momentum γ is added to the hyperparameters.
To set the range in which to search for each hyperparameter, we follow the configuration of Choi et al. (2019),Shallue et al. (2019). We summarize the workloads we use in the experiment in Table 2. We do not use batch normalization and the input image is just normalized and any data augmentation
is not employed. Hyperparameter ranges are summarized in Tables 3,4,5 and 6. We conduct a bayesian optimization to explore hyperparameters in the range described in these tables.

heading: C.3 DISTRIBUTION OF TRAIN LOSS AND GENERALIZATION GAP

heading: D ADDITIONAL EXPERIMENTAL RESULTS

heading: D.1 FULL RESULTS OF CORRELATION BETWEEN GENERALIZATION GAP AND TIC LOWER BOUND ESTIMATES
text: We summarize these results, evaluated with three different correlation coefficients, in Table 7. These results are the calculated 3 types of correlation coefficients for the plot shown in figure 4a, 4b and 2c . Furthermore, the relationship between these correlation coefficients and the values of the ratios of the parameters to the number of data is shown in Figure 8. The result of plotting these results along with d/n is shown in Figure 8.

heading: D.2 ADDITIONAL RESULTS OF SMALL-SCALE EXPERIMENTS
text: In this chapter, we provide details of the experimental results on a small-scale that could not be included in the main paper. In particular, we investigate the goodness of approximation of the information matrix in the small-scale case, since it can be computed exactly, although the execution time is longer.

heading: D.2.1 EMPIRICAL RELATIONSHIP BETWEEN H AND F
text: First of all, we examine the behavior ofH(θ) when it is approximated by F (θ), or more precisely, F1mc(θ). Figure 9 shows that H(θ) is not in perfect agreement with F (θ) due to the effect of the damping term. However, For cases that do not require inverse calculations, such as trace calculations and lower bounds, these effects can be eliminated and a relatively good approximation can be achieved. Furthermore, in the case of linear neural networks, H(θ) and F (θ) show a strong correlation.

heading: D.2.2 EFFECT OF MATRIX SHAPE APPROXIMATION ON ESTIMATION OF TIC
text: Next, we fixed the use of F (θ) instead of H(θ) and observed the change in the TIC estimates and the correlation with the generalization gap when form of the matrix is changed by approximation methods.
Figure 10 and 11 show the correlation between the TIC estimates and the generalized gaps in the case of approximation. Figure 10 shows the comparison of the shape of matrix and its estimates, and Figure 11 shows the correlation between generalization gap and tic estimates. These values are evaluated using three different correlation metric and summarized in Table 8, 9 and 10.

heading: D.3 ADDITIONAL RESULTS OF PRACTICAL-SCALE EXPERIMENTS
text: In this section, we present results of practical-scale setting that we have not presented in the main paper. We observed that in a small-scale setting, DNNs with a large number of layers and SC tend to have a high rank correlation in the TIC estimator and a strong correlation with the generalization gap. Since it is not computationally feasible to compute TIC in practical DNNs, this section details experimentalresults on the performance of two approximate estimators of TIC using diagonal approximation and its lower bound.

heading: D.3.1 PRACTICAL-SCALE EXPERIMENTS: MNIST CASE
text: As shown in Figure 12, we observed the correlation between the TIC estimator and the generalization gap for the diagonal approximation and the lower bound. Both were found to be highly correlated and good estimators of the generalization gap.

heading: 100 101
text: Here, we also investigated whether the estimated TIC from lower bounds is due to only one of the components of the trace, respectively. As shown in figure 13, the value of trace itself was not correlated with the generalization gap. It was also confirmed that different models behaved differently. At the same time, we also confirmed that Tr (F (θ)) is a good approximation of trace Tr (H(θ)).

heading: D.3.2 PRACTICAL-SCALE EXPERIMENTS: CIFAR10 AND CIFAR100 CASE

heading: D.3.3 CORRELATION BETWEEN TIC ESTIMATES AND GENERALIZATION GAP IN TRAINING PROCESS
text: Within the scope of our experiments, we find that TIC can estimate the generalization gap even in the middle of learning for problem settings that are considered to belong to NTK.

heading: D.4 CALCULATION TIME MEASUMENTS EXPERIMENT
text: First of all, we explain the execution time of the small NN/LNN case. In the case of using Tr (F (θ)) and Tr (C(θ)), the computation time by block diagonal approximation is only 40% faster than exact computation. The computational complexity should have been reduced from O(d3) to O(d3l ) by the block diagonal approximation. However, the overhead such as memory copying is dominant, and there is no significant difference in execution time compared to the theoretical amount of computation. From block diagonal to diagonal approximation, the experimental results show a reduction of up to 50% in computation time. The computational complexity is reduced from O(d3l ) to O(d).
Secondly, the use of Tr (H(θ)) and Tr (C(θ)) requires up to 900% more time than the use of Tr (F (θ)) and Tr (C(θ)) in the Exact case. The choice to use Tr (F (θ)) instead of Tr (H(θ)) is therefore justified in terms of the reduction in computational time. In the case of block diagonalization, the speed-up was only a few percent. In the case of diagonalization, no significant speed-up was observed, but the speed-up was more than 40 times when Trace’s approximation was performed using the Hutchinson method. However, the estimation using Tr (F (θ)) and Tr (C(θ)) is faster in the Small Scale problem setting.
It is observed that in such a small scale problem setting, there is a 50 times difference in execution time between using Tr (H(θ)) and Tr (C(θ)) with Exact TIC and computing Tr (F (θ)) and Tr (C(θ)) simultaneously and using diagonal approximation.
This speed-up should be more significant for larger models. As mentioned in section 2.2, it was not feasible to calculate of Tr (H(θ)) in ResNet-8 which requires more than 2,00 TB of memory without approximation. While execution time is also important, the most important aspect is that it is made possible to calculate TIC by approximation.
Secondly, we compare the execution time at practical scale. Among the cases where Tr (F (θ)) and Tr (C(θ)) is used, we investigate how much the time can be reduced when Tr (F (θ)) and Tr (C(θ)) are computed simultaneously compared to the case where they are computed separately. In the case of Small Scale and Practical Scale networks, the time is reduced by half, but in the case of SimpleCNN, VGG16, ResNet8, etc., the time is not reduced significantly. For relatively small models, which could be reduced by about 50%, the approximation by simultaneous Tr (F (θ)) and Tr (C(θ)) calculations was faster than using the Hutchinson method. In contrast. For networks with a large number of dimensions in the final layer, such as VGG16, etc., the calculation of Tr (C(θ)) and a fast approximation of Tr (H(θ)) resulted in a speedup of 10%-25% faster than the simultaneous calculation of Tr (F (θ)) and Tr (C(θ)).
Ta bl
e 11
:F ul
lD et
ai ls
of R
un tim
e M
ea su
m en
ts E
xp er
im em
nt .U
ni t:
se co
nd
D at
as et
M od
el E
xa ct
w /F
C B
lo ck
D ia g w /F C
D ia g w /F C
L ow
er B
ou nd
w /F
C E
xa ct
w /H
C B
lo ck
D ia g w /H C
D ia g w /H C
L ow
er B
ou nd
w /H
C
L ow
er B
ou nd
w /H C by
H ut
ch in
so n’ s M et ho
d Ti ny M N IS T 2- w id e N N 10 .9 09 3 6. 75 16 3. 72 73 1. 90 69 10 5. 42 82 96 .0 64 7 92 .9 79 82 .0 08 1 2. 78 14 Ti ny M N IS T 3- N N 3. 86 41 3. 84 35 3. 79 37 1. 99 4 23 .9 08 3 22 .4 05 5 21 .7 70 5 22 .4 20 3 2. 57 05 Ti ny M N IS T 3- N N w /S C 3. 89 99 3. 85 68 3. 82 39 2. 02 25 26 .2 30 3 24 .9 56 1 25 .0 15 1 24 .9 21 7 2. 89 32 Ti ny M N IS T 3- L N N 3. 86 56 3. 84 7 3. 79 91 1. 95 21 23 .1 88 22 .2 68 5 22 .1 64 1 21 .6 51 1 2. 86 45 Ti ny M N IS T 3- L N N w /S C 3. 87 71 3. 89 62 3. 81 59 1. 95 94 23 .8 57 1 22 .8 10 6 23 .0 33 8 22 .0 72 1 2. 71 98 M N IS T 6- N N N /A N /A 3. 72 35 1. 91 72 N /A N /A N /A N /A 2. 89 49 M N IS T 6- N N w /S C N /A N /A 3. 68 13 1. 94 59 N /A N /A N /A N /A 3. 36 79 M N IS T 6- L N N N /A N /A 3. 48 22 1. 91 48 N /A N /A N /A N /A 2. 80 89 M N IS T 6- L N N w /S C N /A N /A 3. 50 2 1. 93 29 N /A N /A N /A N /A 2. 93 82 M N IS T Si m pl e C N N N /A N /A 4. 70 59 2. 63 37 N /A N /A N /A N /A 8. 79 58 C IF A R 10 6- L N N N /A N /A 3. 67 42 1. 92 95 N /A N /A N /A N /A 4. 10 43 C IF A R 10 6- L N N w /S C N /A N /A 3. 70 03 1. 94 53 N /A N /A N /A N /A 2. 89 96 C IF A R 10 R es N et 8 N /A N /A 12 .4 67 7 10 .0 03 1 N /A N /A N /A N /A 90 .9 63 3 C IF A R 10 V G G 16 N /A N /A 13 .3 07 7 11 .3 72 1 N /A N /A N /A N /A 60 .7 16 C IF A R 10 0 R es N et 8 N /A N /A 12 .4 29 2 9. 96 55 N /A N /A N /A N /A 88 .7 42 3
D.5 EXPERIMENTS WITH d/n CHANGES WITHIN TINYMNIST
We created a restricted dataset, SmallTinyMNIST, which uses only 5% of the TinyMNIST data.
We conducted training evaluations using SmallTinyMNIST to investigate the relationship between the TIC estimator and the generalization gap for patterns with relatively large d/n ratios.
Furthermore, we prepared a network with a large number of d as a 3-layer Wide-NN, and confirmed the effectiveness of TIC in settings where the d/n ratio exceeds 10.
The distributions of the loss and the generalization gap are shown in figure 18. For the comparative purpose, problem settings with the same or relative networks are included in comparison plots and tables. The correlation between the TIC and the generalization gap is shown in figure 19.

Review summaries:
The authors experimentally investigate the ability of Takeuchi's information criterion to predict generalization gaps in various neural network architectures, with a particular focus on the Neural Tangent Kernel (NTK) regime. This was previously done only on small scale networks [1], and the authors extend it to state-of-the-art large scale architectures. In order to do this successfully, the authors propose to use various approximations, including the following:  (1) The equivalent representation of the expected Hessian matrix H via the Gauss-Newton Matrix  (with the equivalence proven in [2]).  (2) A block diagonalization approach that ignores correlations between layers.  (3) An even coarser approximation that consists in ignoring the off-diagonal elements of the relevant matrices H (expected hessian) and C (correlaction of the gradient) (4) Hutchinson's method to estimate the trace of the Hessian (from [3])   The authors run a vast number of experiments and demonstrate that Takeuchi's information criterion exhibits significant correlation with the generalization gap and that many of their approximations perform well. The authors also propose to use the TIC to improve hyperparmeter search, with encouraging results.       / ========== References ==========  [1] Valentin Thomas, Fabian Pedregosa, Bart van Merrie ̈nboer, Pierre-Antoine Mangazol, Yoshua Ben- gio, and NL Roux. Information matrices and generalization. 2019 arXiv preprint  (Appears to have in fact been published at AiStats 2020  with the title " On the interplay between noise and curvature and its effect on optimization and generalization". )   [2] James Martens. New insights and perspectives on the natural gradient method. JMLR 2020    [3] Haim Avron and Sivan Toledo. Randomized algorithms for estimating the trace of an implicit sym- metric positive semi-definite matrix. Journal of the ACM 2011 

This paper focuses on a fascinating, challenging problem: evaluating the generalization performances of neural networks based on the training set with some criteria. Specifically, the authors propose to measure the generalization ability via Takeuchi’s Information Criteria (TIC). Besides, the authors approximate the TIC to accelerate the process of estimation. The proposed method is promising and works well when d/n is large (where d = #parameters and n = #samples).  However, I am still puzzled about some details. I will raise my score if the authors could clarify those concerns (see the following).

The paper empirically studies the effectiveness of Takeuchi’s information criteria (TIC) for deep neural networks near the NTK regime. Since exact computation for TIC (Tr(H^{-1} C)) is often intractable for practical DNNs, the authors explore various approximations of TIC.   The authors’ claimed contributions are: - Empirical and theoretical evidence that TIC is highly correlated with the generalization gap in DNNs that are close to the NTK regime. Given TIC is not designed for singular models, this is a surprising finding. - Conduct comprehensive experiments (varying models, datasets, training settings) showing TICs estimate generalization gaps - Application of these findings to prune poorly performing models in hyperparameter optimization demonstrates this can successfully prevent promising candidates being pruned prematurely.