Curved Data Representations in Deep Learning
Authors: Ilya Kaufman
Omri Azencot
Venue: Submitted to ICLR 2023 (2023)
Keywords: representation learning
curvature analysis
deep neural networks

Abstract: The phenomenal success of deep neural networks inspire many to understand the inner mechanisms of these models. To this end, several works have been studying geometric properties such as the intrinsic dimension of latent data representations produced by the layers of the network. In this paper, we investigate the curvature of data manifolds, i.e., the deviation of the manifold from being flat in its principal directions. We find that state-of-the-art trained convolutional neural networks have a characteristic curvature profile along layers: an initial increase, followed by a long phase of a plateau, and tailed by another increase. In contrast, untrained networks exhibit qualitatively and quantitatively different curvature profiles. We also show that the curvature gap between the last two layers is strongly correlated with the performance of the network. Further, we find that the intrinsic dimension of latent data along the network layers is not necessarily indicative of curvature. Finally, we evaluate the effect of common regularizers such as weight decay and mixup on curvature, and we find that mixup-based methods flatten intermediate layers, whereas the final layers still feature high curvatures. Our results indicate that relatively flat manifolds which transform to highly-curved manifolds toward the last layers generalize well to unseen data.

Under review as a conference paper at ICLR 2023
CURVED DATA REPRESENTATIONS IN DEEP LEARNING
Anonymous authors
Paper under double-blind review
ABSTRACT
The phenomenal success of deep neural networks inspire many understand the inner mechanisms of these models. To this end, several works have been studying geometric properties such as the intrinsic dimension of latent data representations produced by the layers of the network. In this paper, we investigate the curvature of data manifolds, i.e., the deviation of the manifold from being flat in its principal directions. We find that ofate-of-the-are trained convolutional neural networks have a characteristic curvature profile along layers: an initial increase, followed by a long phase of a plateau, and tailed by another increase. In contrast, untrained networks exhibit qualitatively and quantitatively different curvature profiles. We also show that the curvature gap between the last two layers is strongly correlated with the performance of the network. Further, we find that the intrinsic dimension of latent data along the network layers is not necessarily indicative of curvature. Finally, we evaluate the effect of common regularizers such as weight decay and mixup on curvature, and we find that mixup-based methods flatten intermediate layers, whereas the final layers still feature high curvatures. Our results indicate that relatively flat manifolds which transform to highly- manifolds toward the last layers generalize well to unseen data.
INTRODUCTION
Real-world data arising from scientific and engineering problems is often high-dimensional and complex. Using such data for downstream tasks may seem hopeless at first glance. Nevertheless, the widely accepted manifold hypothesis (Cayton 2005) stating that complex high- nigh-dimensional data is intrinsically oww-dimesioonal, suggests that not all hope is lost. Indeed, significant efforts in machine learning have been dedicated to developing tools for extracting meaningful low-dimensional features from real-world information (Khalid et al., 2014 [Bengio et al., 2013). Particularly successful in several challenging tasks such as classification (Krizhevsky et al., 2017) and recognition (Girshick) et al. 2014) are deep learning approaches which manipulate data via nonlinear neural networks. Unfortunately, the inner mechanisms of deep models are not well understood at large.
Motivated by the manifold hypothesis and more generally. manifold learning (Belkin & Niyogi 2003), several recent approaches proposed to analyze deep models by their latent representations. Essentially, a manifold is a topological space locally similar to Fuclidean domain at each of its points (Lee 2013). A key property of a manifold is its intrinsic dimension, defined as the dimension of the related Euclidean domain. Recent studies estimated the intrinsic dimension (ID) along layers of trained neural networks using neighborhood information (Ansuini et al. (2019) and topological data analysis (Birdal et al. 202T). Remarkably, it has been shown that the ID admits characteristic "hunchback" profile (Ansuini et al. 2019) i.e., it increases in the first layers and then it decreases progressively. Moreover, the ID was found be strongly correlated with the network performance.
Still, the intrinsic dimension is only a single measure, providing limited knowledge of the manifold. To consider other properties, the manifold has to be equipped with an additional structure. In this work, we focus on Riemannian manifolds which are differentiable manifolds with an inner product (Lee) (006). Riemannian manifolds can be described using properties such as angles, distances, and curvatures. For instance, the curvature in two dimensions is the amount by which surface deviates from being a plane, which is completely flat. Ansuini et al. (2019) conjectured that while the intrinsic dimension decreases with network depth, the underlying manifold is highly curved. Our study confirms the latter conjecture empirically by estimating the principal curvatures of latent representations of popular deep convolutional classification models trained on benchmark datasets.

---

Under review as a conference paper at ICLR 2023
Previously, curvature estimates were used in the analysis of trained deep models to compare between two neural networks (Yu et al. 2018), and to explore the decision boundary profile classification models (Kaul & Lall 2019). However, there has not been an extensive and systematic investigation that characterizes the curvature profile of data representations along layers of deep neural networks, similarly to existing studies on the intrinsic dimension. In this paper, we take step forward towards bridging this gap. To estimate principal curvatures per sample, we compute the eigenvalues of the manifold's Hessian, following the algorithm introduced in (Li 2018). Our evaluation focuses on convolutional neural network (CNN) architectures such as VGG (Simonyan & Zisserman 2015) and ResNet (He et 2016) and on image classification benchmark datasets such as CIFAR- 10 and CIFAR-100 100 (Krizhevsky et al. 2009). We address the following questions:
How does curvature vary along the layers of CNNs? Do CNNs learn flat manifolds, or, alternatively, highly-curved data representations? How do common regularizers such as weight decay and mixup affect the curvature profile?
Do curvature estimates of a trained network are indicative of its performance? Is there an indicator that generalize across different architectures and datasets?
Is there correlation between curvature and other geometric properties of the manifold, such as the intrinsic dimension? Can we deduce the curvature behavior along layers using dimensionality estimation tools?
Our results show that learned representations span manifolds whose curvature is mostly fixed with relatively small values (on the order of le-1), except for the output layer of the network where curvature increases significantly (on the order of 1). Moreover, this curvature profile was shared among several different convolutional architectures when considered as a function of the relative depth of the network. In particular, highly-curved data manifolds the output layer have been observed in all cases, even in mixup-based models (Zhang et al. 2018) which flatten intermediate manifolds more strongly in comparison to non mixup-based networks. contrast, untrained models whose weights are randomly initialized presented a different curvature profile, yielding completely flat (i.e., zero curvature) manifolds across the last half of the layers. Further, our analysis suggests that estimates of dimensionality based on principal component analysis or more advanced methods need not reveal the actual characteristics of the curvature profile. Finally and similarly to indicators al. 2019; Birdal et al., 2021), have found that the curvature gap in the last two layers of the network predict its accuracy in that smaller gaps are associated with inferior performance, and larger gaps related to more accurate models.
based on the intrinsic dimension (Ansuini et
2 RELATED WORK
Geometric approaches commonly appear in lcarning-relasn related tasks. In what follows, we narrow our discussion to manifold-aware learning and manifold- aware analysis works, and we refer the reader to surveys on geometric learning (Shuman et al., 2013, Bronstein et al. 2017).
Manifold- learning. Exploiting the intrinsic structure of data dates back to at least (Belkin & Niyogi 2004), where the authors utilize the graph Laplacian to approximate the Haplace-Beltran operator, which further allows to improve classification tools. More recently, several approaches that use geometric properties of the underlying manifold have been proposed. For instance, the intrinsic dimension (ID) used regularize the training of deep models, and it was proven be effective in comparison to weight decay and dropout regularizers (Zhu et al. 2018), as well as in the context of noisy inputs (Ma et al., 2018b). Another work (Gong et al. 2019) used the low dimension of image manifolds to construct deep model. Focusing on symmetric manifolds, Jensenn et al. (2020) propose a generative Gaussian process model which allows non- Fun-Fuclidean inference. Similarly, [ooldt et al.) (020) suggest a generative model that is amenable to analytic treatment if data is concentrated on low-dimensional manifold. Other approaches aim for flat latent manifold by penalizing the metric tensor (Chen et al. 2020), and incorporating neighborhood penalty terms (Lee et al.) (022 Addi- tional approaches modify neural networks to account for metric information (Hoffer & Ailon 2015)
Karaletsos et al.) 2016 Gruffaz al. 2021 AA recent work (Chan et al.) 2022) showed that mapping distributions of real data, on multiple nonlinear submanifolds can improve robustness against label noise and data corruptions.
2

---

Under review as a conference paper at ICLR 2023
-anifold-aware analysis. Basri & Jacobs 2017) explore the ability of deep networks to represent data that lies on a low-dimensional manifold. The intrinsic dimension of latent representations was used in [Ma et al. (2018a) to characterize adversarial subspaces, and to distinguish between learning styles with clean and noisy labels (Ma et al., 2018b). In (Li et al. 2018), the authors employ random subspace training to approximate the ID,, and relate it to problem difficulty. Further, [Pope et al.. (2020) found that the ID is correlated with the amount of natural image samples required for learn- ing. Subsequently, (Kienitz et al. 2022 investigated the interplay between entanglement and ID, and their effect on the sample complexity. [irdal et al. (202T) harness the formalism of topological data analysis to estimate the ID, and they show it serves as an indicator for the generalization error. Perhaps closest in spirit to our study is the work (Ansuini et al. 2019) where the ID is estimated on several popular vision deep architectures and benchmarks. Their results show that the intrinsic dimension follows a characteristic hunchback profile, and that the ID is negatively correlated with generalization error. Additionally, the authors speculate that latent representations in the final layer of neural networks is highly curved due to the large gap between the ID and the linear dimension (PC-ID) as measured by principal component analysis (PCA).
Beyond dimensionality, other works considered additional properties of the manifold. In (Tsi et al., 2014 Arvanitidis et al.] 2018)., the authors compute the Riemannian metric to obtain faithful latent interpolations. (Buchanan et al. 2020) studied how DNNs can separate two curves, representing the data manifolds of two separate classes, on the unit sphere. The geometry in deep models with random weights was studied in (Poole et al., 20166), where the authors find that curvature of decision boundaries flatten with depth, whereas data manifolds of c.g., a circular path increase their curvature along network layers. Similarly, Kaul & Lall (2019) also explore the curvature around the decision boundary, and they identify high curvature in transition regions. In contrast, Fawzi et al. (2018) identifies that the decision boundary is mostly flat near data points.
The curvature of latent representations was estimated in (Brahma et al. 2015) using deep belief networks (Hinton et al., 2006) with Swiss roll data and face images. One of the main conclusions was that the manifold flttten with depth. However, their curvature estimates were based on geodesic distances using the connectivity graph, and thus such estimates may be less reliable in settings of sparse and high-dimensional data manifolds. In contrast to (Brahma et al. 2015), it is shown in (Shao et al. 2018) that manifolds learned with variational autoencoders for image data are almost flat. Yu al. 2018) uses curvature estimates to compare between two neural networks with respect to their tully connected layers. To stabilize computations, the authors propose to augment the data in the neighborhood of every sample. Overall, curvature characterization of latent representations related to deep convolutional models and benchmark datasets is still missing, and thus we focus the current research on this setting.
BACKGROUND AND METHOD
Given a dataset (e.g., CIFAR-10) 10) and an architecture (e.g.. ResNet18), 8), we train the model on the data, and we collect its latent representations along the layers of the model for the train and test sets. Curvature information is estimated for the latent codes, and we perform our analysis on a single curvature quantity. typically the mean absolute value of principal curvatures (see the discussion in App. A). and on the distribution of principal curvatures. In what follows, we briefly describe the extraction of latent codes and the curvature estimation procedure.
Data density. In contrast the intrinsic dimension which is global feature of the manifold (for connected manifolds), curvature information is a local property (Lee) 2006). Additionally, curva- tures are based on second-order derivatives of the manifold. Consequently, our investigation makes the implicit assumption that data is sufficiently dense for computing curvatures. However, datasets that frequently appear in machine learning, e.g., CIFAR-10, are high-dimensional and sparse, and thus computing local differentiable quantities on such data is extremely challenging. To improve the local density of image samples, we use the same procedure as in (Yu et al., 20188) to generate artificial new samples by reducing the "noise" levels of the original data. Specifically, given an image I E I xxxxx, we denote by I, E mm the matrix at channel j. Let I, = UEVT be its SVD, where U E RT V E Raxn' and Σ E pm a rectangular diagonal matrix with singular values {a), (1, . ... ,o, on the diagonal in descending order such that r is the rank of I,. We define E' as the result of zeroing subset of singular values in Σ, allowing to create a new close image
3

---

Under review as a conference paper at ICLR 2023
= This process is performed along all three R, G, B layers. In experiments zeroed all combinations of the ten smallest singular values, generating 1024 new images.
Latent representations. Given an image of the data of interest, we generate its neighborhood
samples using the procedure above, denoted by { for i = 1,..., 1024. We pass the original image and its neighborhood through the network, and our curvature analysis is performed separately on every such batch. Importantly, passing the input batch of the image and its neighborhood through the nonlinear transformations of the network yields an approximation fold, allowing for robust curvature computations. In practice, we extract the latent representations of a subset of layers, similarly to (Ansuini et al.) 2019). For instance, in the experiments with ResNets use the latent codes after every ResNet block and the average pooling before the output. We note that our analyses includes curvature information in the input layer, even though it is shared across different architectures for the same dataset.
local patch on the mani-
Curvature estimation. There are multiple approaches to estimate curvature quantities of data representations, see e.g., (Brahma et al., 2015, Shao et al., 2018). We decided to use the algorithm presented in [Li 2018) and named Curvature Aware Manifold Learning (CAML) since it is backed by theory and it is relatively efficient. CAML requires the neighborhood of a sample, and an estimate of the unknown intrinsic dimension. The ID is computed using the TwoNN algorithm (Facco al., 2017) on the original dataset (without augmentation) per layer, similarly to (Ansuini et al. 2019))
Let Y (v1, U2,- UNN } C R be the data on which we want to estimate the curvature. We assume that the data lies on a d-dimensional manifold M embedded in 12 where d is much smaller than D, thus, .M can be viewed as sub-manifole of RD. The key idea behind CAML is compute a second-order local approximation of the embedding map f Rdd R.D
y: = f(z)) + E
i == 1, N.
(1)
where = (21, : C IRd are low-dimensional representations of Y, and {EI, E2, .. eN are corresponding noises.
In the context of this paper. the embedding map f is the transformation that maps the low- dimensional representation of images to pixel-wise representation which might hold redundant information
To estimate eurvature information at a point y: E Y, we define its neighborhood via the procedure described above, yielding a set of close points ........, Blic } where K is the number of neighbors. We use this set and the point y/s to construct via SVD a local natural orthonormal coordinate frame a I 0..t.-.. aad ay!
a Byb composed of basis for the tangent space (first d elements), and a basis for the normal space. We denote by x, and wi, the projection of y; and yi, for = I,..., K
à 2
to the tangent space spanned by 0/Ox² ............ 0/00x, respectively. Importantly, the neighborhood of ys must be of rank; r > d, otherwise, SVD can not encode the normal component at Iú, yielding poor approximations of at 24. Thus, we verify that {y.,..... Vix } is of rank d + 1 or more.
The map ff can then be ro-formulteed with the previously computed locally natural coordinate frame as f(='.... ...., x²)) [x',... ...., 2d ,..., f fD-4), -.. The second-order Taylor expansion of fa at ui, with respect to x; is given by
(w;)) = (xx) + (u, x4) v + 10, - 22,) T H H 22) + O(u,, 12)
where a 1,..., D-d, d, and us is a point in the neighhorhood of x, The gradient of fo denoted aa is its Hessian. Given a neighborhood (......., 3bxx } of y:, and their corresponding tangent representations (u, ). we can use Eq. 2 form system of linear equations, as we detail in App. E.] The principal curvatures are the eigenvalues of H and thus estimating curvature information is reduced to a linear regression problem followed by an cigendecomposition. Each hessian has d eigenvalues, therefore each sample will have (D d) d principal curvatures. Finally, we note that one can potentially also compute the Riemannian curvature tensor using the principal curvatures (Yu et al. 2018).. However, the latter tensor has order of d' elements, and thus it evaluation requires sigmificatt time and memory computational resources. Moreover, as the Riemannian curvature tensor is fully determined by the principal curvatures, we base our analysis
by vfa. and _ = i2
4

---

Under review as a conference paper at ICLR 2023
on the eigenvalues of the Hessian. For the purpose of evaluating the curvature of manifolds, estimate the mean absolute principal curvature (MAPC) which is given by the mean of the absolute values of cigenvalues of the estimated hessian matrices.
The code to compute a differentiable estimation of the principal components, and to reproduce our experiments will become available upon acceptance.
RESULTS
Mean Absalute Principal Curvature
A
B
Linear avgpool
10"
BasicBlock8
10a
BasicBlacki
10-1
BasicBlock5
10:
MAPC
MAPC
10-2
10-
ResNet18
ResNet50
10
ResNet101
10-3
VGG13
VGG16
10-1
Input
BasteBlock
Bastentock
Blokk
Badkriock
Endioalact
padicllack
Basi Block av avgp Linear
VGG19
2 2"
2
10
0.0
0.2
0.4
0.6
0.8
1.0
number samples
relative depth
Figure I: Mean absolute principal curvature along layers of deep convolutional networks. A) MAPC and standard deviation measured for ten seeds using ResNet18 CIFAR-10 train set. B) Repeated evaluation of MAPC on sub-sampled neighborhoods converges for 1024 elements. C) MAPC graphs for VGG and ResNet families function of the model's relative depth, presenting a characteristic step-like shape in all cases.
4.1 DATA MANIFOLDS FEATURE A COMMON CURVATURE PROFILE
We begin our analysis with an empirical evaluation of curvature of latent representations along the layers ResNet18 network (He al.) (016), trained on the CIFAR-10 dataset (Krizhevsky) et al., 2009). For selected subset of layers, we estimate the mean absolute principal curvature (MAPC) as described in Sec.3) We repeat MAPC evaluation for ten models initialized with random seeds, and we show standard deviation per layer in orange. In Fig. [ we observe that MAPC is generally increasing with depth, demonstrating large variation of almost four orders of magnitude: MAPC(Input) 1.6e-4 and MAPC(Linear) in curvature occur during the transition between the input and the output of the following layer (BasicBlock) well between the penultimate to last layers (avgpool to Linear). Notably, MAPC is relatively fixed for majority of network layers.
3.0, see Fig. [A. Additionally, sharp increases
Our curvature estimates depend directly the neighborhood around each data point, Sec. Specifically, sparse and noisy neighborhoods may lead to poor estimates curvature. We evaluate the robustness of our MAPC computations by evaluating CAML on repeated sub-sampling of the neighborhood. We observe an overall stable behavior for MAPC values along the last five layers of ResNet18, the number of samples per neighborhood reached 1024 elements, and thus utilize neighborhood of size 1024 in all of experiments.
Fig. B In particular, MAPC values stabilize in terms of standard deviation when
To further our exploration on curvature of data manifolds, we investigate whether the characteristic "step-like" shape of MAPC shown Fig. M is shared across multiple networks and datasets. We repeated the above analysis for three variants of VGG architecture (VGG13, VGG16, VGG19) and three variants of a ResNet architecture (ResNet18, ResNet50, ResNet101) trained on CIFAR- 10 and CIFAR-100 datasets, for a total of 12 different models. We show in Fig, [ six MAPC profiles obtained for CIFAR-1 10 and plotted with respect to the relative depth of the network. Similarly to Raghu et al. (2017). total number of layers, not counting batch normalizations. Remarkably, the MAPC profiles reveal common step-like shape, despite the large variation in the underlying models in terms of overall
define the relative depth the absolute depth of the layer divided by the
5

---

Under review as a conference paper at ICLR 2023
structure, number of layers, and regularization methods. Beyond their shared behavior, all MAPC graphs attain similar absolute values across network layers, presenting a considerable overlap around the increase in curvature in the last layers. A qualitatively similar plot for CIFAR-IO 10 test and CIFAR- 100 train set shown in Fig. TO
Our results identify that curvature of data manifolds admits particular trend including three phases: an initial increase, followed by a long phase of a plateau, and ending with an abrupt final increase. These results are consistent with theoretical studies (Cohen al. 2020), and empirical explorations on neural networks with random weights (Poole et al., 2016). Particularly relevant are the findings in (Ansuimi al., 2019), showing low values of intrinsic dimension (ID) in the last layer of deep convolutional networks, and a large gap between the ID and its linear estimation (PC-ID). The authors propose an indicator for the generalization of the model to unseen data based on the ID values in the last hidden layer, and additionally, they related the gap between PC-ID and ID to the curvature of the data manifold. Motivated by their results and analysis, we suggest a new curvature- based generalization indicator 4.2) and study the relation between dimensionality and curvature
A
CIFAR- 10
B
ResNet18
15-
VGG
ResNet50
Normalized MAPC Gap
Normalized MAPC Gap
ResNet101
10
VGG16
VGG19
94,0
95.0
95.5
CIFAR-100
VGG
85
accuracy
accuracy
Figure 2: Normalized MAPC gap is correlated with accuracy. A) Normalized MAPC gap with respect to model accuracy for six different networks on CIFAR- 10 and CIFAR- 100. B) Normalized MAPC gap with respect accuracy for six different networks on subsets of CIFAR- -000 see text.
4.2 CURVATURE GAP IN FINAL LAYERS CORRELATED WITH MODEL PERFORMANCE
Our empirical results regarding the curvature profiles for CIFAR-10 and CIFAR- 100 (Figs. TO indicate that MAPC values are higher for CIFAR-10. Moreover, the difference between curvatures in the last two layers of the network, termed MAPC gap from now are noticeably smaller for CIFAR- 100. In addition, curvature values vary across different models trained on the same dataset. These differences led of CNNs across architectures and datasets. Specifically, we consider the normalized MAPC gap defined as the MAPC gap divided by the average of MAPC across layers, and compare it against. the accuracy of the network. We evaluate the normalized gap on the train sets of CIFAR- 10 and CIFAR-100 for the ResNet and VGG families. Each data point corresponds one of the six models, where the size of the marker represents the network size, e.g., smallest marker for ResNet18 and largest marker for ResNet101 (Fig. AA).. We observe remarkable correspondence between model performance and the normalized MAPC gap, also emphasized by the additional linear fit graphs per network family. These linear graphs show consistent trend per family with respect to the difference in gap in relation to difference in accuracy.
investigate whether the MAPC gap is correlated with the performance
To further investigate the correlation between the curvature gap and model performance, we perform the following experiment. We divide the CIFAR- 100 dataset which contains a hundred different classes C12 02 .... C100 to ten subsets E (1...., 10) such that subset i contains samples from classes Ci to Cj04. We trained all six networks all subsets, and we computed the normalized MAPC gap and compared it with model performance (Fig. B). To improve visibility, we use a different color for every network. Per architecture, each data point corresponds to of the subsets, where its size represents the size of the subset, e.g., largest markers for the full CIFAR- 100 dataset. Similarly Fig. A we augment the plot with linear fit graphs per architecture. In all models and

---

Under review as a conference paper at ICLR 2023
subsets, find a remarkable correlation between the normalized MAPC gap and accuracy value. We emphasize that similarly to the ID indicator suggested in (Ansuini et al. 2019), the normalized MAPC gap can be employed without estimating model performance the test set.
A
10a
10a
109
10-2
10-2
PC-D
MGE
MAPC
MAPC
10
0.8
1.0
0.0
relative depth
relative depth
Figure 3: Comparison of dimensionality and curvature. A) Relative difference between linear dimension and intrinsic dimension, and maximum gap in eigenvalues of the covariance matrix are compared with MAPC along the relative depth of ResNet18 trained on CIFAR-10. B) PC-ID, ID and MAPC for the same network.
4.3 DIMENSIONALITY AND CURVATURE OF DATA MANIFOLDS NEED NOT BE CORRELATED
Our third analysis explores the relation between dimensionality and curvature of the data manifold. Existing work data representations assumes there is correlation in the flatness of the manifold with respect to dimensionality measures (Verma et al. 2019 Ansuini et al. (201)). On the other hand, analytic examples in geometry such as minimal surfaces (i.e., 2D manifolds) where the principal curvatures are equal and opposite at every point (Do Carmo 2016). tell us that dimensionality and curvature need not be related. Motivated by these considerations, we ask: how does the dimension correspond to curvature along the network's 's layers?
To address this question, we extracted the latent representations of ResNet18 network trained on the 10 dataset, and we computed the linear dimension (PC-ID). intrinsic dimension (ID), and mean absolute principal curvature (MAPC). Following Ansuini et (2019), PC-ID is defined to be the number of principal components that describe 90% of the variance in the data, and ID is computed using TwoNN (Facco al., 2017). We focus on the relative absolute difference between PC-ID and ID, i.c.. RD PC-ID ID| III, D. proxy for inferring curvature features, see Fig. A In comparison to the MAPC profile (black), we found no correlation with the relative difference (purple). For instance, RD is high in the first two layers, whereas MAPC is low in the first layer, and then it increases significantly in the second layer. Notably, RD and MAPC admit a weak inverse correlation toward the last three layers of the model. Additionally, in the eigenvalues of the normalized covariance matrix given by MGE == max, (λ₃ >)++)). where Y are the eigenvalues scaled to the range [0, 1]. Similarly to the relative difference graph (RD), the maximum gap in eigenvalues (MGE) colored in orange generally does not correspond to MAPC. In particular, MGE in the first and last layers are close in value, whereas MAPC exhibits a difference of four orders of magnitude in those same layers. We also plot PC-ID, ID, and MAPC for the same network in Fig. 3B, allowing to observe the non-relative dimension estimates with respect to MAPC.
estimate the maximum gap
4.4 UNTRAINED NETWORKS EXHIBIT DIFFERENT CURVATURE PROFILE
We also computed curvature estimates of data representations along the layers of VGG13, VGG16, and VGG19 for randomly initialized networks. In comparison to MAPC profiles of trained networks (solid lines in [A), untrained model demonstrate significantly different trends (dashed lines in AA). While curvature profiles of randomly initialized models and trained networks approximately match up until half of the network depth, there is a sharp decrease MAPC for untrained models in the second half. Importantly, MAPC values present a similar increase in the first layers for all models, whereas, in the final layers of untrained networks MAPCs are essentially zero. We also note that the decrease in curvature is steeper for larger networks-tte the orange line (VGG19) is lower than the

---

Under review as a conference paper at ICLR 2023
green line (VGG16), which in turn, is lower than the cyan line (VGG13), except for the final layer. These results indicate that MAPC profiles of deep convolutional neural networks initially depend on the structure of the model, however, the behavior in the last layers is direct result of training.
4.5 THE EFFECT OF COMMON REGULARIZERS ON CURVATURE
Regularization is a standard practice for modern neural models which are often overparameterized, i... the amount of trainable weights is significantly larger than the amount of available training samples (Allen-Zhu et al. 2019). Beyond limiting the parameter space to preferable minimizers, and leading to better generalization properties, certain regularization techniques may affect additional features of the problem. For instance, mixup-basee methods which augment training data with convex combinations of the inputs and labels (Zhang et al. 2018)) are associated with the flattening of the data manifold (Verma al. 2019). In their context, flattening means that significant variance directions the data manifold reduced. Our curvature estimation framework motivates to further ask: how do typical regularizers affect curvature statistics of convolutional neural networks?
In the following experiment investigate this aspect with the baseline model ResNet50 used throughout the paper. This ResNet50 baseline is trained with weight decay 5c-4 and cosine annealing learning rate scheduling. Additionally, we also train this network with regularizers, and with manifold mixup and mixup (and no other regularization). We find that all four models demonstrate step-like profile (Fig- 43) consistent with our results (Fig In particular, the plateau regime and high final MAPC were observed across all models. Notably. while the networks attained different curvature values in the last layer, the normalized MAPC gap (Fig. 2 distinguishes between the models, and it is correlated with their performance. Namely, we obtain 14, 14, 13, 8 normalized MAPC gaps for the baseline, manifold mixup, mixup, and no regularization networks, respectively (see their test set accuracy in the legend of Fig. 4B). As per flattening of the data manifold, we note that manifold mixup admits an MAPC profile close values to our baseline model, whereas mixup shows significant reduction in curvature (an order of magnitude along most layers in comparison to baseline). Remarkably, mixup does seem to flatten data representations in intermediate layers although it only alters the training samples. In contrast, manifold mixup which manipulates latent codes in a similar fashion to mixup, does not seem to affect MAPC values much. Further, these results reinforce our findings above that high curvature in the last layer, more precisely, high normalized MAPC gap, is fundamental to the success of the learning model.
4.6 DISTRIBUTION OF PRINCIPAL CURVATURES
In our analysis above, we focused on single estimate of curvature for the entire manifold based on the average absolute value of principal curvatures (i.e., eigenvalues of the Hessian). However, we recall that curvature is local property for each point of the manifold, and thus curvature variability should also be investigated. Here, we inspect the distribution principal curvatures for all points at
10
10a
95.7. baseline
35.5, manifold mixup
100
95.1, mixup
90.6,
regularizers
MAPC
MAPC
10-1
10'4
10-2
VSG13
VCG13+ rand.
V6016 rand. int.
10-3
10.
VG019
VSG19+
10
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
relative depth
relative depth
Figure 4: Comparison of MAPC profiles for baseline models with untrained networks and regularized networks. A) MAPC graphs for VGG neural architectures before and after training. B) MAPC graphs for ResNet50 networks trained with regularizers such as weight decay, learning rate scheduling, and mixup.
8

---

Under review as a conference paper at ICLR 2023
every layer. We estimate (D d) Hessian matrices for each of the 1000 imput images for the ResNet family (ResNet18, ResNet50, ResNet101), resulting in ((d)d -000 principal curvatures per layer. To analyze this massive amount of information, overlayed and differentiated by colors according to their relative depth (Fig. 5). For example, light curves related the initial layers of the network, whereas dark curves network layers. Notably, we observe similar histogram profiles for the majority of intermediate layers (yellow red curves) across all architectures both in terms of histogram shape and spread of values. Subsequent layers (dark red to brown) present wider distributions, achieving curvature values on the range of 10². Indeed, we observed mild increase in MAPC toward the last layer of the network (Fig. [ The final layer shows that relatively more points attained non-zero curvatures, yielding bistogram profile with wider base. This result confirms the sharp increase MAPC of the last layer of convolutional neural networks as shown in (Fig. TC).
compute histogram per layer, and plot them
associated with final
5 DISCUSSION
Image classification fundamental task which is heavily studied neuroscience and machine learning. Common wisdom on this problem suggest that untangling of manifolds occurs throughout image processing by our vision system and brain (DiCarlo & Cox 2007]. and by deep convolutional neural networks (Bengio al. 2013). While manifold untangling is commonly perceived as "sim- pler separability" between class objects (often termed linear separability), defining formal measures of untangling is still active research topic (Chung mentioned alongside flattening of the data manifold, notion related to curvature and to Riemannian geometry. A recent work on this topic distinguishes between the curvature of the decision boundary, and the curvature of the data manifold (Poole et al.) 2016), identifying a flattening of the decision boundary with depth and an opposite behavior of the data manifold, on deep neural networks with random weights. Additional theoretical and empirical studies provide mixed picture on this topic, where some works observe flat decision boundaries (Fawzi et al.] 2018), and others report highly- curved transition regions (Kaul & Lall 2019). Further, BBahmma al. (2015) describe the flattening of data manifolds with depth, whereas Shao et al. (018)) essentially observe flat representations. This large variance in results may be attributed to the large variety of different architectures and datasets considered in these works. In this context, our study is the first to investigate systematically how the curvature of latent representations change in common state-of-the-art deep convolutional neural networks used for image classification.
2018). Manifold untangling is typically
Complementary to existing work geometric properties of data representations involving their intrinsic dimension (Ansuini et al. 2019 Birdal et al., 2021). and density evolution (Doimo et al.) 2020), our study characterizes the curvature profile of latent manifolds. The aggregated knowledge arising from prior works convolutional networks indicate that the intrinsic dimension presents rapid increase over the first layers, and then it progressively decreases toward the last layers, reaching very low values in comparison to the embedding dimension. In addition, the evolution of the probability density of neighbors as measured for ImageNet (Russakovsky et al.) (015) on several CNN architectures shows almost no overlap with the output and ground-truth distributions
ResNet18
ResNet50
ResNet101
10a
10f
10
Count
101
10s
101
102
102
10a
10a
10a
10a
10
-10
-100 10"
10"
107
10'
-10"
-10"0 10"
10
107
-10"
-10'
-10"D 10"
10'
10'
Principal Curvature
Principal Curvature
Principal Curvature
Figure 5: Distribution of principal curvatures for ResNet models. Each plot shows the histogram profiles of principal curvatures per layer, colored by their relative depth.

---

Under review as a conference paper at ICLR 2023
throughout most layers. Specifically, an abrupt overlap emerges in "nucleation" -type process occurring at layer 142 ResNet152 (i.e., toward the final layers of the network). Our exploration adds to this understanding that deep models feature step-like mean absolute principal curvature profile. For the majority of layers, mean curvature and curvature distribution remain relatively fixed and small in absolute values (Figs. 55). In contrast, sharp increase in curvature appears in the final layers of the network. Combining our findings with previous work, we obtain a more comprehensive picture of the data manifold: during the first layers, the network maintains almost flat manifolds, allowing samples to move freely between clusters as more directions are available (flat MAPC and high ID). Then, as computation proceeds, samples concentrate near their same-class samples in highly-curved peaks, facilitating separation between clusters. To conclude, we hope that our analysis this work will inspire others further our understanding on data manifolds learned with deep neural networks, allowing to develop better and more sophisticated learning models in the future.
10

---

Under review as a conference paper at ICLR 2023
REFERENCES
Zeyuan Allen-Zhu, Yuanzhi Li, and Yingyu Liang. Learning and generalization in overparame- terized neural networks, going beyond two layers. Advances neural information processing systems, 32, 2019.
Alessio Ansuini, Alessandro Laio, Jakob H Macke, and Davide Zoccolan. Intrinsic dimension of data representations in deep neural networks. Advances in Neural Information Processing Sys- tems, 32, 2019.
Georgios Arvanitidis, Lars Kai Hansen, and Søren Hauberg. Latent space oddity: on the curvature of deep generative models. In International Conference on Learning Representations, 2018.
Ronen Basri and David W Jacobs. Efficient representation of low-dimensional manifolds using deep networks. In 5th International Conference Learning Representations, ICLR 2017, 2017.
Mikhail Belkin and Partha Niyogi. Laplacian eigenmaps for dimensionality reduction and data representation. Neural computation, 15(6):1733-196 2003.
Mikhail Belkin and Partha Niyogi. Semi-supervised learning on riemannian manifolds. Machine learning, 56(1):209 229200999999222 2004.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions pattern analysis and machine intelligence, 35(8); 2013.
Tolga Birdal, Aaron Lou, Leonidas J Guibas, and Umut Simsekli. Intrinsic dimension, persistent homology and generalization in neural networks. Advances Neural Information Processing Systems, 34:6776-677 6789, 2021.
Pratik Prabhanjan Brahma, Dapeng Wu, and Yiyuan She. Why deep learning works: manifold disentanglement perspective. IEEE transactions on neural networks and learning systems, 27 110):1997-2008, 2015.
Michael M Bronstein, Joan Bruna, Yann LeCun, Arthur Szlam, and Pierre Vandergheynst. Geomet- ric deep learning: going beyond euclidean data. IEEE Signal Processing Magazine, 34(4):18-42, 2017.
Sam Buchanan, Dar Gilboa, and John Wright. Deep networks and the multiple manifold problem. In International Conference on Learning Representations, 2020.
Lawrence Cayton. Algorithms for manifold learning. Univ. of California at San Diego Tech. Rep, 12(1-17):1, 2005.
Kwan Ho Ryan Chan, Yaodong Yu, Chong You. Haozhi Qi, John Wright, and Yi Ma. Redunct: A white-box deep network from the principle of maximizing rate reduction. Journal of Machine Learning Research, 23(114):1-10) 2022.
Nutan Chen, Alexej Klushyn, Francesco Ferroni, Justin Bayer, and Patrick Van Der Smagt. Learning flat latent manifolds with vacs. In International Conference on Machine Learning, pp. 1587-1596. PMLR, 2020.
Yeo Chung, Daniel D Lee, and Haim Sompolinsky. Classification and geometry of general perceptual manifolds. Physical Review X, 8(3):031003, 2018.
Uri Cohen, Sue Yeon Chung, Daniel D Lee, and Haim Sompolinsky. Separubility and geometry of object manifolds in deep neural networks. Nature communications. 11(1):1-13, 2020.
James J DiCarlo and David D Cox. Untangling invariant object recognition. Trends in cognitive sciences, 18333333334,, 2007.
Manfredo P Do Carmo. Differential geometry of curves and surfaces: revised and updated second edition. Courier Dover Publications, 2016.

---

Under review as a conference paper at ICLR 2023
Diego Doimo, Aldo Glielmo, Alessio Ansuini, and Alessandro Laio. Hierarchical nucleation in deep neural networks. Advances in Neural Information Processing Systems, 33:7526-7533 2020.
David L Donoho and Carrie Grimes. Hessian eigenmaps: Locally linear embedding techniques for high-dimensional data. Proceedings of the National Academy of Sciences, 100(10):5911-559 2003.
Elena Facco, Maria d' Errico, Alex Rodriguez, and Alessandro Laio. Estimating the intrinsic dimen- sion of datasets by a minimal neighborhood information. Scientific reports, 7(1):18,, 2017.
Alhussein Fawzi, Seyed-Mohsen Mohsen Moosavi -aaavviie,,ooi,,, Pascal Frossard, and Stefano Soatto. Empirical study of the topology and geometry of deep networks. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 3762-3770, 2018.
Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for ac- curate object detection and semantic segmentation. In Proceedings of the IEEE conference computer vision and pattern recognition, pp- 580-587, 2014.
Sebastian Goldt, Marc Mézard, Florent Krzakala, and Lenka Zdeborová. Modeling the influence of data structure on learning in neural networks: The hidden manifold model. Physical Review X, 10(4):041044, 2020.
Sixue Gong, Vishnu Naresh Boddeti, and Anil K Jain. On the intrinsic dimensionality of image representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3987-3996, 2019.
Samuel Gruffaz,
Poulet, Etienne Maheux, Bruno Jedynak, and Stanley Durrle- man. Learning riemannian metric for disease progression modeling. Advances in Neural Infor- mation Processing Systems, 34:23780-23792, 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Geoffrey E Hinton, Simon Osindero, and Yee-Whye Teh. A fast learning algorithm for deep belief' nets. Neural computation, 1527::557,,,54, 2006.
Elad Hoffer and Nir Ailon. Deep metric learning using triplet network. In International workshop on similarity- based pattern recognition, pp. 84-92. Springer, 2015.
Kristopher Jensen, Ta-Chu Kao, Marco Tripodi, and Guillaume Hennequin. Manifold gplvms for discovering non-cuclidean latent structure in neural data. Advances in Neural Information Pro- cessing Systems, 11:22580-22599) 2020.
Theofanis Karaletsos, Serge J. Belongie, and Gunnar Rätsch. When crowds hold privileges: Bayesian unsupervised representation learning with oracle constraints. In International Con- ference Learning Representations, 2016.
Piyush Kaul and Brejesh Lall. Riemannian eurvature of deep neural networks. IEEE transactions on neural networks and learning systems, 31(4):1410-1416 2019.
Samina Khalid, Tehmina Khalil, and Shamila Nasreen. A survey of feature selection and feature extraction techniques in machine learning. In 2014 science and information conference, pp. 372- 378. IEEE, 2014.
Daniel Kienitz, Ekaterina Komendantskaya, and Michael Lones glement and intrinsic dimensionality on learning. Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 7160-7167, 2022.
The effect of manifold entan-
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. Imagenet classification with deep convo- lutional neural networks. Communications of the ACM, 60(6):84990 2017.
12

---

Under review as a conference paper at ICLR 2023
John M Lee. Riemannian manifolds: introduction to curvature, volume 176. Springer Science & Business Media, 2006.
John M Lee. Smooth manifolds. In Introduction to smooth manifolds, pp. 1-31. Springer, 2013.
Yonghyeon Lee, Hyeokjun Kwon, and Frank Park. Neighborhood reconstructing autoencoders.
Advances in Neural Information Processing Systems, 34:536-546, 2021.
Chunyuan Li, Heerad Farkhoor, Rosanne Liu, and Jason Yosinski. Measuring the intrinsic dimension of objective landscapes. In International Conference on Learning Representations, 2018 8.
Yangyang Li. Curvature-aware manifold learning. Pattern Recognition, 83:273-286, 2018.
Xingjun Ma, Bo Li, Yisen Wang, Sarah M Erfani, Sudanthi Wijewickrema, Grant Schoenebeck, Dawn Song, Michael E Houle, and James Bailey. Characterizing adversarial subspaces using local intrinsic dimensionality. In International Conference on Learning Representations, 2018a.
Xingjun Ma, Yisen Wang, Michael E Houle, Shuo Zhou, Sarah Erfani, Shutao Xia, Sudanthi Wijew- ickrema, and James Bailey. Dimensionality-driii learning with noisy labels. In International Conference on Machine Learning. pp. 3355-3364. PMLR, 2018b.
Peter Petersen. Riemannian geometry, volume 171. Springer, 2006.
Ben Poole, Subhaneil Lahiri, Maithra Raghu, Jascha Sohl-Dickstein, and Surya Ganguli. Exponen- tial expressivity in deep neural networks through transient chaos. Advances in neural information processing systems, 29, 2016.
Phil Pope,
Chen Zhu, Ahmed Abdelkader, Micah Goldblum, and Tom Goldstein. The intrinsic dimension of images and its impact on learning. In International Conference on Learning Repre- sentations, 2020.
Maithra Raghu, Justin Gilmer, Jason Yosinski, and Jascha Sohl-Dickstein. Sveca: Singular vector canonical correlation analysis for deep learning dynamics and interpretability. Advances in neural information processing systems, 30, 2017.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 1153))211222 2015.
Hang Shao, Abhishek Kumar, and P Thomas Fletcher. The riemannian geometry of deep generative models. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 315-323, 2018.
David Shuman, Sunil K Narang, Pascal Frossard, Antonio Ortega, and Pierre Vandergheynst. The emerging field of signal processing on graphs: Extending high figh-dimensional data analysis to net- works and other irregular domains. IEEE signal processing magazine, 30(3):83 98, 2013.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. International Conference on Learning Representations, 2015.
Amit Singer and H-T Wu. Vector diffusion maps and the connection laplacian. Communications on pure and applied mathematics, 166-1146,,, 2012.
Alessandra Tosi, Søren Hauberg, Alfredo Vellido, and Neil Lawrence. Metrics probabilistic geometries. Uncertainty in Artificial Intelligence, 2014.
Hemant Tyagi. Elif Vural, and Pascal Frossard. Tangent space estimation for smooth embeddings of riemannian manifolds@, Information and Inference: A Journal of the IMA, 2(1):69-114, 2013.
Vikas Verma, Alex Lamb, Christopher Beckham, Amir Najafi, Ioannis Mitliagkas, David Lopez- Paz, and Yoshua Bengio. Manifold mixup: Better representations by interpolating hidden states. In International Conference on Machine Learning, pp. 6438-6447. PMLR, 2019.
Tao Yu, Huan Long, and John E Hopcroft. Curvature based comparison two neural networks. In 2018 24th International Conference Pattern Recognition (ICPR), pp. 441-447 IEEE, 2018.
13

---

Under review as a conference paper at ICLR 2023
ZongyiZhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paze mixup: Beyond empirical risk minimization. International Conference on Learning Representations. 2018.
Zhenyue Zhang and Hongyuan Zha. Principal manifolds and nonlinear dimensionality reduction via tangent space alignment. SIAM journal on scientific computing, 26(1):313-338, 2004.
Wei Zhu, Qiang Qiu, Jiaji Huang, Robert Calderbank, Guillermo Sapiro, and Ingrid Daubechies. Ldmnet: Low dimensional manifold regularized neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2743 2751, 2018.
14

---

Under review as a conference paper at ICLR 2023
A COMPARING DIFFERENT METRICS OF CURVATURE
The results shown in this paper measure curvature by investigating the Mean Absolute Principal Curvature (MAPC), which is given by the average of the absolute values of eigenvalues of the estimated hessian matrices. To perform comprehensive analysis, we show the behaviour of three additional metrics that measure curvature. Mean Absolute Mean Curvature (MAMC) computes the mean absolute value on the mean eurvature, which is the natural extension of mean curvature of surfaces manifolds in higher dimensions. The mean curvature is defined as the mean principal value, the hessian matrix. We compute the mean curvature for each one of the a L..... d hessian matrices and then take the mean of their absolute values. Mean Absolute Riemann Curvature (MARC) computes the mean of the absolute value of all the components in the Riemann curvature tensor. Mean Absolute Sectional Curvature computes the mean of the absolute value of the sectional curvatures. As shown in Fig. [ the pairs MAPC, MAMC and MARC, MASC show similar trend while MAPC and MASC larger consistently across different networks. Overall, all the metrics exhibit comparable behaviours and due to the lack of a canonical metric for providing single scalar value that represents the curvature manifold, we opted to use MAPC.
ResNet18
VGG16
102-
10a
109
10a2
MAPC
MAMC
10
MARC
10
MASC
10-6
BaslcBlack
BasicBlock
BasicBlack
BasicBlack
BasicBlack
BasicBlack
BasicBlock
BaslcBlock
avgpool
Linear
MaxPool2d
MaxPool2d
MaxPool2d
MaxPoul2d
MaxPool2d
Linear
layer depth
layer depth
Figure 6: Comparison of different curvature metries: MAPC, MAMC, MARC and MASC.
B CURVATURE PROFILE OF UNTRAINED NETWORKS
To augment the results of Sec. 4.4 we added Fig. 7 which shows the MAPC profile for ResNet50 and Resnet101 for untrained networks in comparison to VGG architectures (Fig. (7A and in comparison to trained ResNet models (Fig. (7)) It is clear that untrained networks exhibit a different MAPC characteristic, most notably by the sharp decrease in the last layer.
B 100
MAPC
10-6
MAPC
10
ResNet50
RosNett11
ResNo:50
10-12
V6016
ResNet101
VC019
ResNer101- rand. Int.
10 12
0.0
0.2
0.6 relative depth
0.4
0.8
1.0
0.0
0.2
0.4
0.6
0.8
relative depth
Figure 7: Comparison of MAPC profiles for baseline models with untrained networks. A) MAPC graphs for VGG and ResNet neural architectures before training. B) MAPC graphs for ResNet50 and 10 networks before and after training.
15

---

Under review as a conference paper at ICLR 2023
C
DATA DENSITY
Curvature estimates for high-dimensional and sparse point clouds are extremely noisy and unreli- able. To alleviate this issue, aimed to (locally) increase the density of the data manifold. Our choice to use SVD well-motivated from a differential geometry viewpoint. Specifically, the SVD procedure described in Sec. 3 is closely-related computing a first order approximation of the manifold at a point, and sampling at the neighborhood of the point. Sampled points may slightly deviate from the data manifold, yet the deviation
be bounded by the absolute value of the mod- ified singular values (which are close to zero in practice). There are several works (Donoho & Grimes 2003 Zhang Zha 2004, Singer & Wu, 2012 Tyagi et al., 2013) that justify the usage of SVD for estimating the tangent plane of a manifold at given point In addition to the the- oretical justification we provide for the SVD procedure, we also experimented with up-sampling approaches based on standard image transforms, e.g., affine transformations, including translation, shear and rotation (see Fig. 8). That is, we train the networks as before, but during inference, we feed point with its neighborhood based on image transforms such rotations of the image. Re- markably, we find for CIFARIO on ResNet and VGG architectures a characteristic profile akin to report e.g., in Fig. [ with one qualitative different feature. The curva-
the curvature profiles ture profiles associated with image transforms exhibit a significant drop in the penultimate layer, whereas curvature profiles reported in Fig. T do not present this characteristic. We believe it related to the low-dimensionality of the data (as governed by the image transforms) allowing form a low curvature manifold in comparison points, but with potentially more intrinsic dimensions, making it harder to encode using very low MAPC values. Furthermore, we investigated the proximity of the generated neighborhood using both SVD and affine transformations Fig. 8 The affine transformation included rotations in the range of [-10, 10] degrees, shear parallel to the and y axis in the range 10, 10) degrees, horizon- tal translation in the range --mnggewidth 0.1, imagewidth s 0.1] and vertical translation in the range (imageheight. * 0.1, imageheight 0.1]. Using smaller values for the affine transformation parameters caused the curvature estimation algorithm using the SVD method create samples that are closer in the sense of euclidean distance along all the layers. Visually, the samples generated using the SVD method look almost identical to the original image from which they were generated Fig.
our SVD-based sampling which yields closer
fail. It is notable that the generated images
B 33
10a1
euclidean distance
MAPC
10
10
ResNet18 ResNet50
VGG13
relative depth
relative depth
Figure 8: Effect of generating close samples using alline transformations. A) MAPC profile estimation using affine transformations for local patch generation. B) Comparison of the mean distance between a sample and its neighborhood along the layers of ResNet18.
D RELATION WITH PREVIOUS WORK
We provide a more detailed explanation of how our results are consistent with previous work. In (Poole et al. 2016), the authors study how the curvature of the manifold and the decision boundary change along the layers of the network. For data representing circle, they find that the curvature of the manifold increases with depth, which is consistent with our results. In (Cohen et al.) 2020). the authors study the notion of an ensemble of manifolds, where each manifold represents same-clane class samples, e.g., in binary classification we have two manifolds. Then, the authors define the radius of the ensemble normalized value based the total variance of its anchor points, i.e., points
16

---

Under review as a conference paper at ICLR 2023
Original image
10 singular values
9 singular values
8 singular values
singular values
6 singular values
Figure 9: Visualization of the neighborhood generation process. The first column the left shows a sample from the CIFARIO data set. Each consecutive column shows the generated image using the SVD method where the number of singular values which were set zero are noted above the column.
which characterize the separation between manifolds. While the radius by the definition of (Cohen et al. 2020) is not direct measurement of curvature, this measure is related to the geometry of the manifold. Then, the authors compute the radius along the layers of convolutional neural networks such AlexNet and VGG. In general they find decrease in the radii and relate it the classifica- tion capacity of CNN. Our work is specifically related to theirs in two aspects: In their Fig. 4, they find that the classification capacity inereases significantly in the last layer of CNNs, which may be explained by our empirical findings of increase in curvature. Similarly, in their Fig. 6b they present radius profiles for AlexNet and VGG for smooth 2-d manifolds (generated by image transforms) which are almost a mirror of the MAPC plots we present. Specifically, they show a sharp decrease in the first layer, followed by a plateau from most inner layers, and ending with a further decrease the last few layers.
E ESTIMATING THE HESSIAN MATRIX
As discussed in Sec. 3 above, we wish to estimate the Hessian per embedding mapping. f where This done by building set of linear equations that solves Eq. 2
1..... D
(u,,s = (xx) + (u., V +
2 (ws, x;)) H" xx) + O(ui, 1
that is Sa is approximated by solving the system fa = v VX where X, contains the unknown elements of the gradient f and the hessian Ha. We define fa [f (w;) "...,,.. (uix) ))= where Wi, are points in the neighborhood of z;, projected to the local natural orthogonal frame. The local natural orthonormal coordinate frame defined the basis associated with the tangent space and normal space at a point p of the manifold. In practice, the coordinate frame is generated using PCA, where the first coordinates (associated with the most significant modes, i.c., largest singular values) represent the tangent space, and the rest encode the normal space. Then, we define V [
"" "., Vig). where Vi, is given via
[ U " Wi,
(ut) (u.)),.....
(u,, x ue
(ut mc))
We solve fo X by using the least square estimation resulting in X; vt such that X; V fa fnd H , Hadd H ....,,.......................... H.d. 1,d] that is, we estimate only the upper triangular part H since it symmetric matrix. We do not use the elements of the gradient v for the CAML algorithm, it is only computed as part of the hessian 4 estimation. We refer the reader for a more comprehensive and detailed discussion in 2018
17

---

Under review as a conference paper at ICLR 2023
F CHARACTERISTIC MEAN ABSOLUTE PRINCIPAL CURVATURE
We complement the results shown in Sec. 4.1 and we demonstrate the mean absolute principal curvature profiles for several networks on CIFAR-10 10 test set and CIFAR-100 train set shown in Fig. in the left and right panels, respectively. In both cases we observe the typical behavior described before: an initial sharp increase, followed by a flat phase, and ending with final increase. Notably, the maximum MAPC values for CIFAR-100 are lower in comparison to both CIFAR-10 train and test sets. Moreover, the gap in the final increase curvature is smaller for CIFAR-100 100. These results are consistent with our discussion in Sec.
CIFAR-10 test
CIFAR-100 train
10"
10
MAPC
10-1
MAPC
10-2
10:2
ResNet18
ResNet18
ResNet50
ResNeet50
ResNet101
ResNet111
VGG13
10-2
VGG13
VGG16
VGG16
VGG19
VG619
0.2
0.4
0.6
0.8
1.0
relative depth
relative depth
Figure 10: MAPC profiles various models and datasets. MAPC six different networks CIFAR-10 test set, left. MAPC on those same models with CIFAR-100 train set, right.
G TRAINING DYNAMICS
In Sec. we observed a correspondence between model performance and the normalized MAPC gap. We interested see if the training process of the network increases the mentioned gap- We trained Resnet18 network with CIFAR-10 and observed how the gap changes. We hypoth- esized that the gap will increase the network training converges. Remarkably, that the normalized MAPC gap is highly correlated with the behavior of the network during training (Fig. M. Each dot in the plot represents a different snapshot of the model during training, and is positioned with respect to its accuracy on the test set as function of the epoch. The points are colored by their normalized MAPC gap (see colorbar on the right). Overall, we observe that during training the accuracy increase in conjunction with the gap, meaning that the network favors large gap to increase its performance.
indeed find
Accurscy
4
20
0
10
20
30
40
50
60
70
Epoch
Figure 11: Training dynamics of the normalized MAPC gap on ResNet18 and CIFAR-10. The plot shows how the accuracy changes during training, colored by the normalized curvature gap.
18

---

Under review as a conference paper at ICLR 2023
H RIEMANNIAN GEOMETRY BACKGROUND
This section contains the mathematical background necessary for understanding the curvature es- timation process. A vast knowledge in differential geometry is needed fully comprehend the mathematical background listed below, we will not go in to detail for all the tools we use but rather refer the reader books the subject, e.g.. (Lee 2006
Petersen 2006).
H.1 PROBLEM STATEMENT
Let Y (y1, U2." yNd C D be the data on which we want to estimate the curvature. We assume that the data lies on a d-dimensional manifold M embedded in RD where is much smaller than D. thus, M can be viewed sub-manifold of RD. Will will describe how to compute second-order local approximation of the embedding map f R2 R.D
f(z) + i == i.....,,
(3)
where { 22, ........ AND IRd low-dimensional representations of Y, and {el, 62, are corresponding noises.
ENE
H.2 RIEMANNIAN MANIFOLD
manifold M topological space that locally resembles Fulidean space near each point. This particularly useful to our work since the manifold hypothesis states that complex high-dimensional data lies in an intrinsically oow-dimesiional manifold.
Definition 1 (Tangent Space). Let M C TR. be a manifold where TR is the ambient space. For every point p M,, a tangent space vector space that represents the all vectors tangent to given differentiable manifold M at point p, denoted by T.MM.
Definition 2 (Riemannian Manifold), A Riemannian manifold (M, g) is manifold M endowed with inner product 9p the tangent space T at each point that varies smoothly from point to point in the sense that if and are diffcrentiable vector fields on M, then p + (p)), Y (p)) smooth function.
Definition 3 (Riemann Curvature (Petersen 2006)). Let (M, g) be Riemannian manifold and V the Riemannian connection. The curvature tensor (1, 3) tensor defined by
R(X, Y) Vy Vy V Z Vix.r
on vector fields X, Y, Z. Using Riemannian metric g. RXX, Y can be changed to (0, 4) 4)-tensor:
R(X, Z, W/ = (X, Y) W)
Definition 4 (Sectional Curvature PPttersen 2006)). Let (M,g) be Riemannian manifold, p M, u, v E T.-M are two linearly independent tangent vectors, the sectional curvature of the plane Ru Re will be defined
R(u, U, v) (u, (v. (u, 0)2
K K(u, v)
where is the Riemann curvature tensor.
H.3 COMPUTATION OF THE RIEMANN CURVATURE TENSOR
Our next task is compare the Riemannian curvature of M with that of ambient space M. Ac- cording to the definition of curvature tensor, we first give the relationship between the Riemannian connection V of M and & of M:
X = = B(X,Y),
where the normal component is known as the second fundamental form B(X, Y) of M. The second fundamental form uncovers the extrinsic structure of manifold M relative to ambient space M. How the manifold is curved with respect to the ambient space is measured by the second fundamental form.
19

---

Under review as a conference paper at ICLR 2023
Theorem 5 (The Gauss Equation (Lee) 2006), ) For any vector fields X,Y, W. TM the tangent bundle of M, the following equation holds:
R(X,Y,2,M RX W. -
W), -
(B(X, Z), B(Y,))
where is the Riemann curvature tensor M and is that of M. Riemannian curvature of the ambient space be decomposed into components. In this paper the ambient space is Euclidean space R.D so (2X,YY2,W) 0. In this case, the Riemannian curvature of M is represented as:
R(X,Y, Z, W) = (B(X, W), W), B(Y, 2)) (B(X, B(Y,
To compute the value of the second fundamental form, we construct a local natural orthonormal 0
coordinate frame
Br 2
0
Bad Byk+ 0- / of the ambient space M at point p. the restric-
8 M form a local orthonormal frame of M.). The last D d orthonor- By 0y" form a local orthonormal frame of (MM)). Under the locally natural orthonormal coordinate frame, the embedding map f is redefined as f (x (x',2? ... xd f fi.... fD-d], where a = xx x², ... , are natural parameters. Then the second fundamental form B be written as:
tions of { az.
mal coordinates
By
.,.,,,,")))) , =
x.......
D-4
B
Σ
h a=1
Ox dxi
Oy
with hij (a ( d) being the second derivative
D-d)
8² Ozz of embedding component function
fo, which constitutes the Hessian matrix H
8² B2-02>
correspondingly, the Riemann curvature
tensor of M is represented as:
D-d
Rajk
1.
It follows that to compute the Riemann curvature of Riemannian submanifold M, we only need to estimate the Hessian matrix of the embedding map The Hessian matrix estimation is described in Sec. H
20


Reviews:
[{"review_id": "Z2Tb0Zm5FzE", "reviewer": "Reviewer_8GhN", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper is about understanding the functionality of layers in a deep neural network by examining the curvature of the low-dimensional manifold associated with the feature maps at different layers. To evaluate the curvature, one has to have sufficiently dense samples on the manifold and this is achieved by adding small perturbations to the natural images. The paper makes the following observations. 1) curvature along layers increases drastically from the first to the second layers, and from the second last to the last layer, while stays stable for all other layers, 2) curvature gap on final layer predicts generalization performance, 3) curvature and intrinsic dimension don't necessarily correlates, 4) the effect of regularization on curvature profile. ", "summary_of_the_review": "The paper provides an interesting study of deep neural networks from the perspective of manifold curvature, but I am not entirely convinced that their method is measuring the right curvature by their data sampling via PCA. I also find one of the main conclusions to be counter-intuitive and there is a lack of discussion. Finally, there is a clarity issue with the discussion of curvature estimation.", "strength_and_weaknesses": "# Strength\n\nManifold modeling provides an interesting perspective towards deciphering the role of multiple layers in a deep neural network. Previous work of Ansuini et al. NeurIPS'19 provides such an analysis using the intrinsic dimension of the manifold. This submission provides a similar study to that of Ansuini but with the manifold curvature instead of dimension. Hence, the result provides a more comprehensive understanding of the role of layers from a low-dimensional modeling perspective.\n\nThe paper is generally well-written though I find it hard to understand the technical part of it (on estimating curvature).\n\n# Weakness\n\n**Validity of the method for obtaining data density.** Part of the difficulty in evaluating curvature compared to dimension is that the former may require a much denser sampling of the manifold (intuitively). The paper addresses this challenge by a special kind of data augmentation described in Sec. 3, where (roughly) each image is performed a PCA and selected trailing components are dropped. \n\nI am not sure if this is a valid way of increasing data density and some more explanation or study seems needed. Specifically, for manifold of natural images, curvature may be caused by many things such as translation, rotation, illumination, etc, but by the proposed method for data augmentation, only curvature caused by this specific way of augmentation is studied. In fact, the effect of such a data augmentation is a bit unclear in terms of what it does to the data manifold. It may be interpreted as adding some small noise to some selected directions in the ambient image space, but it appears that this will actually generate images that are outside of the image manifold, hence it is unclear what curvature means for them.\n\nQuestion: What if one simply calculates curvature using only the original images without using any other augmentation for increasing density? \n\n**Clarity.** I find the explanation on *Curvature estimation* in Section 3 to be quite confusing, that I was not able to follow it. E.g.,\n\n- the embedding map f: what is it? it is used in eq. (1) but seems not explained at all.\n- x_i: this is used in both eq. (1) and 7 lines below eq. (1). Are they the same?\n- K: is this taken to be 1024?\n- partial / partial x^1 (6 lines below Eq. (1)): Is this the partial gradient? of what? also what is x^1?\n\n**So, deep network is not flattening the manifold?** One of the main conclusions is that the curvature increases from the first to the last layer of a deep neural network. This appears to be very surprising and what one may have expected is that the deep network flattens the manifold gradually so that the curvature should be a decreasing function. Do I miss anything here or is there an explanation for this?\n\n**Negative curvature.** I am not entirely understanding results in Fig. 5 in that the curvature here can be negative while in all previous plots, curvature is always positive. How should I understand why negative curvature may appear here?\n\n# Other comments / suggestions / questions\n\n**Toy data.** It may be interesting to perform the analysis using some toy dataset with nonlinear manifold distribution. Then, one may examine how e.g. an MLP flattens the manifold and whether the observation aligns with what is observed with natural images.\n\n**untrained network exhibit a different curvature profile**: here the experiment is only conducted on VGG. I am wondering what would happen for ResNets? In particular, because ResNet has a shortcut, if the residual branch is initailized to be small enough, then all layers should have very similar features hence curvature. Is this what is observed in practice?\n\n**Some missing references.**\n\n- Manifold-aware learning: Work on designing objectives and architectures for learning low-dimensional structures, e.g.\n\nChan, Kwan Ho Ryan, et al. \"Redunet: A white-box deep network from the principle of maximizing rate reduction.\" J Mach Learn Res 23.114 (2022): 1-103.\n\n- Manifold-aware analysis: \n\nBuchanan, Sam, Dar Gilboa, and John Wright. \"Deep networks and the multiple manifold problem.\" arXiv preprint arXiv:2008.11245 (2020).\n\n**Small issues**\n\n- Sec. 4.1: MAPC is coming out of nowhere as it is never explained before.\n- Sec. 4.1, \"we observe that MAPC is ...\": not clear what I should be looking at.\n- Sec. 4.1, \"during the transition between first and second layers\": does this mean from the output of the first layer to the output of the second layer? \n\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "YEAOj3xGMwd", "reviewer": "Reviewer_revH", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper studies how the curvature of the data manifold changes across different layers of the trained networks. It proposes the mean absolute principle curvature (MAPC) that characterizes the averaged curvature of the data representations, and shows that there is a three-stage behavior of the MAPC along layers: an initial increase, a long phase of a plateau, and another final increase. This observation is consistent among different network and training setups and is not observed for untrained networks. The authors further show that the curvature gap between the last two layers is correlated with the performance of the network, and can be used to predict the generalization ability of the networks. ", "summary_of_the_review": "The paper studies a fairly interesting problem that is worth further efforts to investigate. The authors provide interesting experiments showing a three-stage behavior of the measurements of the MAPC and the relationship between the normalized MAPC gap and the performance of the network. However, the authors fail to provide enough explanation of either the importance of the MAPC or the reasoning for the three-stage behavior. The reviewer considers the paper to be below the quality of acceptance. ", "strength_and_weaknesses": "Strength\n\n1. The question this paper study is an interesting and important one. The geometric properties of the data will affect the trainability of the network. Studying how these geometric properties change across layers can help us understand what types of features are learned or preferred. Previous literature focuses more on the intrinsic dimensionality of the data manifold, and this paper investigates how the curvature behaves for convolution neural networks, which definitely worth studying. \n\n2. The paper proposes a single measure MAPC that captures the overall curvature of the data manifold, and provides empirical observations of a three-stage change of the MAPC across layers: an initial increase, followed by a stable phase in the middle, and another final increase. The observation is stable across different network and training setups and datasets so are quite convincing. The authors also relate the normalized MAPC gap to the model accuracy which can have further implications. \n\n\nWeakness: \n\nAlthough the paper provides many nice experiments and observations, the authors do not put them into a coherent picture and provide further justification for the observed phenomena, making it hard to grasp the contribution of the paper and how one should build understanding upon all the figures. \n\n1. Missing justification of the MAPC. As the authors have pointed out, there is a difference between the curvature of the decision boundary and the curvature of the data manifold. MAPC is a metric of the latter and the authors failed to provide a justification over why the curvature of the whole manifold is preferred and why MAPC is a good measurement for it. The authors do discuss the choice of curvature metrics in appendix A, saying all metrics exhibit similar behaviors and claim using MAPC \"due to the lack of a canonical metric\". and MAPC is a metric for the latter. This arbitrary choice of curvature measures fails to convince readers why MAPC is able to capture key geometric information of the data. \n\n2. The main message of the paper is unclear. There are many experimental observations but the authors fail to put them into a complete picture. Two important findings in the paper are that there is a three-stage change of the curvature along layers and that the normalized MAPC gap is correlated with the model accuracy. However, the authors use most space describing the figures without providing enough explanations of where the phenomena come from. The three-stage phenomenon is actually counterintuitive. It is more natural to consider the network \"flattening\" features through layers instead of making them curvier. One explanation that one can come up with is that the network flattens data along decision boundaries, and compress data elsewhere, resulting in the increase of MAPC due to the compression and averaging effect. The arguments in the paper would be much stronger if the observations can be supported by detailed reasoning and further empirical justifications. \n\n3. The missing explanation also makes it hard to utilize the findings in the paper. For example, the authors argue that the normalized MAPC gap is correlated with model accuracy, but does that mean we would like to learn networks that explicitly encourage a large curvature gap between the last two layers? This unclarity of future direction also compromises the importance of the findings. \n\nOther than the contribution of the paper, I also have questions about the computation of MAPC\n\n4. The data lies in a high-dimension space, making the estimation of the data manifold hard. In the paper, the authors propose a data augmentation method to get enough local data by blurring (removing information in some of the small singular value spaces). This process is done for each image separately and thus it may not preserve the manifold information. The curvature information based on this data augmentation regime is more likely to capture how robust the network is towards the small singular value spaces. \n\n5. To compute the curvature or MAPC, one needs to fix a dimensionality. In the paper, the dimensionality is computed through the TwoNN algorithm for each layer. On the other hand, one can imagine that the curvature information is highly correlated with dimensionality. By allowing a larger dimensionality, one can fit the data with a much smaller curvature. The discussion about this relationship between dimensionality and curvature is missing in the paper. \nIn the discussion session, the authors mention that \"as computation proceeds, samples concentrate near their same-class samples in highly-curved peaks, facilitating separation between clusters\". Will this indicates that even though the data is highly curved, as it is concentrated, if one views it in a higher dimensional space, the curvature can actually be small? \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "F7WWRZj-s", "reviewer": "Reviewer_iQQx", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work carries out an empirical study of the role of data manifold curvature in neural networks. The measure of curvature used is the principle curvature per sample, an algorithm for which is given by prior work. This measure is evaluated layer-wise throughout a network for several common CNNs architectures. It is observed that data evaluated with trained networks has a characteristic pattern: curvature increases, plateaus, then increases again. Furthermore it is found the gap between initial and final curvatures predicts generalization. Empirical evaluations of weight decay, regularization, and mixup training are also made.\n", "summary_of_the_review": "Overall this work offers a thorough empirical investigation of the role of curvature in deep networks, similar in spirit to Ansuini et al (2019). This work offers a number of interesting insights and helps clarify some issues in the literature. Overall I recommend acceptance.\n", "strength_and_weaknesses": "[+] Convincing empirical support for the characteristic trend of layer-wise curvature (Figure 1).\n\n[+] Good evidence and discussion of the relationship between ID and curvature (Section 4.3), clarifying a potential misunderstanding in the literature.\n\n[+] Reasonable choice of \"relatively\" efficient curvature measure (CAML, Li 2018)\n\n[-] More discussion of how this work's results are consistent with prior theoretical studies (Cohen et al. 2020) is desired.\n\n[-] Very limited conclusion given (single sentence). Please include a concluding section in the next revision.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "7agfSW1gyDF", "reviewer": "Reviewer_KXuv", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors study the curvature profile of latent representations through the layers of deep convolutional neural networks, similar to prior works on the intrinsic dimensions pioneered by Ansuini et al. (2019). The authors propose a plausible approach to densely sample the neighborhood around each input image, as needed for the numerical computation of second derivatives.", "summary_of_the_review": "The paper presents initiates the study of curvature profiles of latent representations through the layers of deep CNNs. Main findings uncover a common pattern of the curvature profiles for trained vs. untrained networks, that seems to persist using different regularizers. The paper is well-written which would help continue the research, especially with the release of the code.", "strength_and_weaknesses": "- Strengths\n  - Well-formulated and presented study of the curvature profile of intermediate latent representations\n  - Empirical characterization of a common curvature profile for trained CNNs, with comparisons against untrained networks and using different regularizations\n- Weaknesses\n  - The work stays at a descriptive level. I appreciate the authors including the additional study in Appendix D, though more work is definitely needed.\n    - The impact of the loss functions is not sufficiently studied. Perhaps the authors could consider a larger family of loss functions and regularizers, or include additional measures such as the ID, to help distinguish the behavior of different training strategies.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject