Year: 2020

Abstract:
We discover that deep ReLU neural network classifiers can see a low-dimensional Riemannian manifold structure on data. Such structure comes via the local data matrix, a variation of the Fisher information matrix, where the role of the model parameters is taken by the data variables. We obtain a foliation of the data domain and we show that the dataset on which the model is trained lies on a leaf, the data leaf, whose dimension is bounded by the number of classification labels. We validate our results with some experiments with the MNIST dataset: paths on the data leaf connect valid images, while other leaves cover noisy images.

heading: 1 INTRODUCTION
text: In machine learning, models are categorized as discriminative models or generative models. From its inception, deep learning has focused on classification and discriminative models (Krizhevsky et al., 2012; Hinton et al., 2012; Collobert et al., 2011). Another perspective came with the construction of generative models based on neural networks (Kingma & Welling, 2014; Goodfellow et al., 2014; Van den Oord et al., 2016; Kingma & Dhariwal, 2018). Both kinds of models give us information about the data and the similarity between examples. In particular, generative models introduce a geometric structure on generated data. Such models transform a random low-dimensional vector to an example sampled from a probability distribution approximating the one of the training dataset. As proved by Arjovsky & Bottou (2017), generated data lie on a countable union of manifolds. This fact supports the human intuition that data have a low-dimensional manifold structure, but in generative models the dimension of such a manifold is usually a hyper-parameter fixed by the experimenter. A recent algorithm by Peebles et al. (2020) provides a way to find an approximation of the number of dimensions of the data manifold, deactivating irrelevant dimensions in a GAN.
Similarly, here we try to understand if a discriminative model can be used to detect a manifold structure on the space containing data and to provide tools to navigate this manifold. The implicit definition of such a manifold and the possibility to trace paths between points on the manifold can open many possible applications. In particular, we could use paths to define a system of coordinates on the manifold (more specifically on a chart of the manifold). Such coordinates would immediately give us a low-dimensional parametrization of our data, allowing us to do dimensionality reduction.
In supervised learning, a model is trained on a labeled dataset to identify the correct label on unseen data. A trained neural network classifier builds a hierarchy of representations that encodes increasingly complex features of the input data (Olah et al., 2017). Through the representation function, a distance (e.g. euclidean or cosine) on the representation space of a layer endows input data with a distance. This pyramid of distances on examples is increasingly class-aware: the deeper is the layer, the better the metric reflects the similarity of data according to the task at hand. This observation suggests that the model is implicitly organizing the data according to a suitable structure.
Unfortunately, these intermediate representations and metrics are insufficient to understand the geometric structure of data. First of all, representation functions are not invertible, so we cannot recover the original example from its intermediate representation or interpolate between data points. Moreover, the domain of representation functions is the entire data domain Rn. This domain is mostly composed of meaningless noise and data occupy only a thin region inside of it. So, even if representation functions provide us a distance, those metrics are incapable of distinguishing between meaningful data and noise.
We find out that a ReLU neural network implicitly identifies a low-dimensional submanifold of the data domain that contains real data. We prove that if the activation function is piecewise-linear (e.g. ReLU), the neural network decomposes the data domain Rn as the disjoint union of submanifolds (the leaves of a foliation, using the terminology of differential geometry). The dimension of every submanifold (every leaf of the foliation) is bounded by the number of classes of our classification model, so it is much smaller than n, the dimension of the data domain Rn. Our main theoretical result, Theorem 3.1, stems from the study of the properties of a variant of the Fisher Information matrix, the local data matrix. However, Theorem 3.1 cannot tell us which leaves of this foliation are meaningful, i.e. what are the possible interesting practical applications of the submanifolds that compose the foliation. The interpretation of this geometric structure can only come from experiments. We report experiments performed on MNIST dataset. We choose to focus on MNIST because it is easily interpretable and because small networks are sufficient to reach a high accuracy.
Our experiments suggest that all valid data points lie on only one leaf of the foliation, the data leaf. To observe this phenomenon we take an example from the dataset and we try to connect it with another random example following a path along the leaf containing the starting point. If such a path exists, it means that the destination example belongs to the same leaf of the foliation. Visualizing the intermediate points on these joining paths, we see that the low-dimensional data manifold defined by the model is not the anthropocentric data manifold composed of data meaningful for a human observer. The model-centric data manifold comprises images that do not belong to a precise class. The model needs those transition points to connect points with different labels. At the same time, it understands that such transition points represent an ambiguous digit: on such points, the model assigns a low probability to every class.
The experiments also show that moving orthogonally to the data leaf we find noisy images. That means that the other leaves of the foliation contain images with a level of noise that increases with the distance from the data leaf. These noisy images become soon meaningless to the human eye, while the model still classifies them with high confidence. This fact is a consequence of the property of the local data matrix: equation (8) prescribes that the model output does not change if we move in a direction orthogonal to the tangent space of the leaf on which our data is located.
This remark points us to other possible applications of the model-centric data manifold. We could project a noisy point on the data leaf to perform denoising, or we can use the distance from the data leaf to recognize out-of-distribution examples.
The main contributions of the paper are: 1. the definition of the local data matrix G(x,w) at a point x of the data domain and for a given model w, and the study of its properties; 2. the proof that the subspace spanned by the eigenvectors with non-zero eigenvalue of the local data matrix G(x,w) can be interpreted as the tangent space of a Riemannian manifold, whose dimension is bounded by the number of classes on which our model is trained; 3. the identification and visualization of the model-centric data manifold through paths, obtained via experiments on MNIST.
Organization of the paper. In Section 2, we review the fundamentals of information geometry using a novel perspective that aims at facilitating the comprehension of the key concepts of the paper. We introduce the local data matrix G(x,w) and we summarize its properties in Prop. 2.1. In Section 3, we show that, through the local data matrix, under some mild hypotheses, the data domain foliates as a disjoint union of leaves, which are all Riemannian submanifolds of Rn, with metric given via G(x,w). In Section 4, we provide evidence that all our dataset lies on one leaf of the foliation and that moving along directions orthogonal to the data leaf amounts to adding noise to data.

heading: 2 INFORMATION GEOMETRY
text: Here we collect some results pertaining to information geometry (Amari, 1998; Nielsen, 2018), using a novel perspective adapted to our question, namely how to provide a manifold structure to the space containing data.
Let p(y|x,w) be a discrete probability distribution on C classification labels, i.e. p(y|x,w) = (pi(y|x,w))i=1,...,C , x ∈ Σ ⊂ Rn, w ∈ Rd. In the applications, x represent input data belonging to a certain dataset Σ, while w are the learning parameters, i.e. the parameters of the empirical model. As we are going to see in our discussion later on, it is fruitful to treat the two sets of variables x and w on equal grounds. This will naturally lead to a geometric structure on a low dimensional submanifold of Rn, that we can navigate through paths joining points in the dataset Σ (see Section 4).
In order to give some context to our treatment, we define, following Amari (1998) Section 3, the information loss I(x,w) = − log(p(y|x,w)) and the loss function L(x,w) = Ey∼q[I(x,w)]. Typically L(x,w) is used for practical optimizations, where we need to compare the model output distribution p(y|x,w) with a certain known true distribution q(y|x). We may also view L(x,w) as the Kullback-Leibler divergence up to the constant − ∑ i qi(y|x) log qi(y|x), irrelevant for any optimization problem:
L(x,w) = Ey∼q[− log(p(y|x,w))] = C∑ i=1 qi(y|x) log qi(y|x) pi(y|x,w) − C∑ i=1 qi(y|x) log qi(y|x) =
= DKL(q(y|x)||p(y|x,w))− C∑ i=1 qi(y|x) log qi(y|x)
(1)
A popular choice for p(y|x,w) in deep learning classification algorithms is
pi(y|x,w) = softmax(s1(x,w), . . . , sC(x,w))i = esi(x,w)∑C j=1 e sj(x,w) , (2)
where s(x,w) ∈ RC is a score function determined by parameters w. From such p(y|x,w) we derive the cross-entropy with softmax loss function:
L(x,w) = Ey∼q[I(x,w)] = Ey∼q[− log p(y|x,w)] = −syx(x,w) + log C∑
j=1
esj(x,w), (3)
where L(x,w) is computed with respect to the probability mass distribution q(y|x) assigning 1 to the correct label yx of our datum x and zero otherwise. Other approaches rely on label smoothing (Szegedy et al., 2016), hence they take a different L(x,w). However, since our treatment mainly relies on the expression of I(x,w), our results will also apply to such loss functions, provided some hypotheses, that we list in Section 3, are satisfied.
Going back to the general setting, notice that: Ey∼p[∇wI(x,w)] = 0. In fact,
Ey∼p[∇w(log(p(y|x,w))] = C∑ i=1 pi(y|x,w)∇w log pi(y|x,w) = C∑ i=1 ∇wpi(y|x,w) = ∇w1 = 0.
(4)
Let us now define the following two matrices:
F (x,w) = Ey∼p[∇w(log(p(y|x,w)) · ∇w(log(p(y|x,w))T ] (5)
G(x,w) = Ey∼p[∇x(log(p(y|x,w)) · ∇x(log(p(y|x,w))T ] (6) We call F (x,w) the local Fisher matrix at the datum x and G(x,w) the local data matrix given the model w. The Fisher matrix (Amari, 1998) is obtained as F (w) = Ex∼Σ[F (x,w)] and it gives information on the metric structure of the space of parameters. Similarly, we can reverse our perspective and see how G(x,w), allows us to recognize some structure in our dataset.
The following observations apply to both F (x,w) and G(x,w) and provide the theoretical cornerstone of Section 3.
Proposition 2.1. Let the notation be as above. Then:
1. kerF (x,w) = span{∇w log pi(y|x,w)|1 ≤ i ≤ C}⊥; kerG(x,w) = span{∇x log pi(y|x,w)|1 ≤ i ≤ C}⊥.
2. rank F (x,w) < C, rank G(x,w) < C.
Proof. See Appendix B.
This result tells us that the rank of both F (x,w) andG(x,w) is bounded byC, the number of classes in our classification problem. The consequence of the bound on rank F (x,w) is the following: in SGD dynamics with a single example, the number of directions in which the change in our parameters modifies our loss is severely limited. The consequence on the bound on rank G(x,w) is even more striking: it will allow us to define a submanifold of Rn of dimension rankG(x,w) (Section 3), that our experiments show contains our dataset (Section 4). In practical situations, this dimension is much lower than the size of G(x,w), i.e. the input size n, as shown in Table 1.

heading: 3 THE MODEL VIEW ON THE DATA MANIFOLD
text: We now turn to examine some properties of the matrices F (x,w) and G(x,w) that will enable us to discover a submanifold structure on the portion of Rn occupied by our dataset (see Section 4) and to prove the claims at the end of the above section.
We recall that, given a perturbation of the weights w, the Kullback-Leibler divergence, gives, to second-order approximation, the following formula:
DKL(p(y|x,w + δw)||p(y|x,w)) = (δw)TF (x,w)(δw) +O(||δw||3) (7)
Equation (7), together with Proposition 2.1, effectively expresses the fact that during SGD dynamics with a mini-batch of size 1, we have only a very limited number of directions, namely C − 1, in which the change δw affects the loss.
Taking the expectation with respect to x ∼ Σ on both sides of the equation, we obtain the analogous property for the Fisher matrix F (w) = Ex∼Σ[F (x,w)]. While F (x,w) has a low rank, the rank of F (x) is bounded by |Σ|(C − 1), a number that is often higher than the size of F (x). Thus F (w), when non-degenerate, is an effective metric on the parameter space. It allows us to measure, according to a certain step δw, when we reach a stable predicted probability and thus the end of model training (see Martens (2020) and refs. within).
It must be however noted that F (w) retains its information content only away from the trained model, that is, well before the end of the training phase (see Achille et al. (2018) for an empirical validation of such statements). We are going to see, with our experiments in Section 4, that a similar phenomenon occurs for G(x,w). Our Fig. 2 elucidates the rank behaviour of G(x,w) during training.
We now turn to the local data matrix G(x,w), thus interpreting equation (7) in the data domain Rn. For a perturbation δx of the data x, we have, up to second order approximation:
DKL(p(y|x+ δx, w)||p(y|x,w)) = (δx)TG(x,w)(δx) +O(||δx||3) (8)
This equation suggests to view G(x,w) as a metric on the data domain. However, because of its low rank (see Proposition 2.1), we need to restrict our attention to the subspace ker(G(x,w))⊥, where G(x,w) is non-degenerate. G(x,w) allows us to define a distribution D on Rn. In general, in differential geometry, we call distribution on Rn an assignment:
x 7→ Dx ⊂ Rn, ∀x ∈ Rn
where Dx is a vector subspace of Rn of a fixed dimension k (see Appendix A for more details on this notion in the general context).
Assume now G(x,w) has constant rank; as we shall see in our experiments, this is the case for a non fully trained model (see Section 4 for more details). We thus obtain a distribution D:
x 7→ Dx = ker(G(x,w))⊥ ⊂ Rn (9)
Equation (8) tells us that, if we move along the directions of ker(G(x,w)), the probability distribution p(y|x,w) is constant (up to a second order approximation) while our data is changing. Those are the vast majority of the directions, since rank G(x,w) < C and typically C << n, hence, we interpret them as the noise directions. On the other hand, if we move from a data point x along the directions in ker(G(x,w))⊥, data will change along with the probability distribution p(y|x,w) associated with it. These are the directions in which the model moves with confidence; we are going to clarify this key point in Section 4.
We now would like to see if our distribution (9) defines a foliation structure. This means that we can decompose Rn as the disjoint union of submanifolds, called leaves of the foliation, and there is a unique submanifold (leaf) going through each point x (see Fig. 7 in Appendix A, where the distribution is generated by the vector field X and R2 is the disjoint union of circles, the leaves of the foliation). The distribution comes into the play, because it gives the tangent space to the leaf through x: Dx = ker(G(x,w))⊥. In this way, moving along the directions in Dx at each point x, will produce a path lying in one of the submanifolds (leaves) of the foliation.
The existence of a foliation, whose leaf through a point x has tangent space Dx, comes through Frobenius theorem, which we state in the Appendix A and we recall here in the version that we need.
Frobenius Theorem. Let x ∈ Rn and letD be a distribution in Rn. Assume that in a neighbourhood U of x: [X,Y ] ∈ D, ∀ X,Y ∈ D. (10) Then, there exists a (local) submanifold N ⊂ Rn, x ∈ N , such that TzN = Dz , for all z ∈ N . It is not reasonable to expect that a general classifier satisfies the involutive property (10), however it is remarkable that for a large class of classifiers, namely deep ReLU neural networks, this is the case, with p given by softmax as in equation (2). Theorem 3.1. Let w be the weights of a deep ReLU neural network classifier, p given by softmax, G(x,w) the local data matrix. Assume G(x,w) has constant rank. Then, there exists a local submanifold N ⊂ Rn, x ∈ N , such that its tangent space at z, TzN = ker(G(z, w))⊥ for all z ∈ N .
Proof. See Appendix B.
Through the application of Frobenius theorem, Theorem 3.1 gives us a foliation of the data domain Rn. Rn decomposes into the disjoint union of C − 1 dimensional submanifolds, whose tangent space at a fixed x ∈ Rn is Dx = ker(G(x,w))⊥ (see Appendix A). Every point x determines a unique submanifold corresponding to the leaf of the foliation through x. We may extend this local submanifold structure to obtain a global structure of manifold on a leaf, still retaining the above property regarding the distribution.
As we shall see in Section 4, we can move from a point x in our dataset Σ to another point x′ also in Σ with an horizontal path, that is a path tangent to Dx, hence lying on the leaf of x and x′. Our experiments show that we can connect every pair of points (x, x′) ∈ Σ×Σ with horizontal paths. It means that all the dataset belongs to a single leaf, which we call the data leaf L. Our model, through the local data matrix G(x,w), enables us to move on the low-dimensional submanifold L, to which all of our dataset belongs. Of course not all the points of L correspond to elements of the dataset;
however as we show in the experiments, on most points of L the model gives prediction compatible with human observers.
We also notice that each leaf of our foliation comes naturally equipped with a metric given at each point by the matrix G(x,w), restricted to the subspace ker(G(x,w))⊥, which coincides with the tangent space to the leaf, where G(x,w) is non-degenerate. Hence G(x,w) will provide each leaf with a natural Riemannian manifold structure. We end this section with an observation, comparing our approach to the geometry of the data domain, with the parameter space.
Remark. Equation (7) provides a metric to the parameter space Rd, motivating our approach to the data domain. For each w ∈ Rd, we can define, as we did for the data domain, a distribution w 7→ D′w := ker(F (w))⊥, using the Fisher matrix. However, it is easy to see empirically that this distribution is not involutive, i.e. there is no foliation and no submanifold corresponding to it.

heading: 4 EXPERIMENTS
text: We performed experiments on the MNIST dataset (LeCun et al., 1998) and on the CIFAR-10 dataset (Krizhevsky et al., 2010). We report in this section the experiments on the MNIST dataset only, because they are easier to interpret in the geometrical framework (foliation and leaves) introduced in our previous section. CIFAR-10 experiments are reported in the dedicated Appendix C. Thus, all the following experiments use a simple CNN classifier trained on the MNIST.
The definition of distribution in Section 3 and the consequent results require the rank of G(x,w) to be constant on every point x for a certain parameter configuration w. We remark that the rank of G(x,w) can be at most C − 1 (Proposition 2), i.e. much lower than the size of G(x,w) (Table 1), so we can expect that rank G(x,w) = C − 1 for almost all x ∈ Rn.
We observe that during the training of the CNN, the rank of G(x,w) is stable and equal to C − 1. That changes at the end of the training, when the model reaches convergence. At that point, the output class probability distribution on most images is highly unbalanced and sparse and all the eigenvalues of G(x,w) tend to 0. Figure 2 shows the decreasing trend of the mean trace of G(x,w) on a batch during the training. The plot is similar to the ones reported in Achille et al. (2018) for the Fisher information matrix and it tells us that the local data matrix G(x,w) loses its informative content at the end of training as well. This observation motivates us to perform subsequent experiments on a partially trained model and with
data points where every output class probability is lower than 0.99.
Since a complex model reaches convergence on MNIST digit classification too fast, we have resorted to a very small CNN. The network used in the following experiments has 18,982 parameters and ReLU activation function. It is similar to LeNet, but with 4 channels in both the two convolutional layers and 32 hidden units in the fully-connected layer. We train this network with SGD for just one epoch to prevent model convergence.
Such a neural network classifier satisfies all the hypothesis of Theorem 3.1, so it views the data domain Rn as a foliation. However, this result does not give us any clue about the characterization of leaves. We know, however, that, moving away from a data point x while remaining on the same leaf, we can obtain points with different labels, while moving in a direction orthogonal to the tangent space to the leaf of x, we obtain points with the same label as x, as long as the estimate (8) holds.
To understand the distinguishing factors of a leaf, we move from a data point x across leaves and we inspect the evolution. We start from an image in MNIST test set and we let it evolve moving orthogonally to leaves. In order to get further and further away, we use a fixed random direction and at every step we project it on the space orthogonal to the current leaf. From the previous section, we know that such a space coincides with kerG(x,w).
We observe in Figure 3, that the noise in the images increases steadily. Moreover, we can see that, as predicted by equation (8), model predictions remain very certain even when the digit is indistinguishable for the human eye.
This experiment makes us speculate that a leaf is characterized by a constant amount of noise. If that is the case, all valid data should reside on the same leaf characterized by the absence of noise, the data leaf. To verify our intuition we follow horizontal paths connecting two images from MNIST test set. If it is possible to join two inputs with a horizontal path, then we know that those points are on the same leaf. In our case a horizontal path is tangent to the distribution D described in the previous section, i.e. Dx = (kerG(x,w))⊥ = span{∇x log pi(y|x,w)|1 ≤ i ≤ C} from Proposition 1. We remark that Dx coincides with the row space of the Jacobian matrix Jacx log p(x,w). We will use this equivalence to efficiently compute the projection of a vector on the distribution Dx. The algorithm to approximate a path from a source point to a destination point is a simplification of Riemannian gradient descent (Gabay, 1982) using the euclidean distance from the destination point as loss function. Since we do not have an explicit characterization of the leaf, we cannot perform the retraction step commonly used in optimization algorithms on manifold. To circumvent this problem, we normalize the gradient vector and use a small step size of 0.1 in our experiments. The normalization assures that the norm of the displacement at every step is controlled by the step size. The iterative algorithm is reported in Algorithm 1.
Algorithm 1: Find a horizontal path between source and destination points. Input: s: source; d: destination; α: step size, T : number of iterations; w: model parameters.
Figure 4 shows the results of the application of Algorithm 1 on some pairs of images in MNIST test set. We observe that it is possible to link very different images with an horizontal path, confirming our conjecture about the existence of the data leaf. This experiment shows that the model sees the data on the same manifold, namely one leaf of the foliation determined by our distribution D. The observation of the paths on the data leaf gives us a novel point of view on the generalization property of the model. First of all, while we could expect to find all training data on the same leaf, it is remarkable that test data are placed on the same leaf too. Furthermore, we observe that the data leaf is not limited to digits and transition points between them. In fact, we can find paths connecting valid images with images of non-existent digits similar to letters, as shown in Figure 5. That fact suggests that the model-centric data manifold is somewhat more general than the classification task on which the model is trained.
Our final experiment gives us another remarkable confirmation of our theoretical findings: it is impossible to link with a horizontal path a noisy image outside the data leaf and a valid data image. Figure 6 shows that even after thousands of iterations of Algorithm 1, it is impossible to converge to the destination point. Indeed, the noise is preserved along the path and the noise pattern stabilizes after 5000 iterations. The final point reached is a noisy version of the actual destination point.
All those experiments confirm that there exists a model-centric data manifold, the data leaf of the foliation. In addition they exhibit that the leaves in the foliation of the data domain are characterized by the noise content.

heading: 5 RELATED WORKS
text: Fisher Information Matrix. In Achille et al. (2018) the authors discuss the information content of the Fisher matrix, and they show that such content changes during the training of a neural network, decreasing rapidly towards the end of it. This shows that the Fisher-Rao metric acquires importance during the training phase only (see also Kirkpatrick et al. (2017)). More on the geometry of the parameter space, as a Riemannian manifold, is found in Sommer & Bronstein (2020). In this paper, we take an analog of the Fisher-Rao metric, but on the data domain. In our examples, we see a phenomenon similar to the one observed in Achille et al. (2018): the trace of the local data matrix decreases rapidly, as the model completes its training. Other metrics on datasets were suggested, for example see Dukler et al. (2019) and refs. within, but with different purposes. Here, our philosophy
is the same as in Bergomi et al. (2019): we believe that data itself is not equipped with a geometric structure, but such structure emerges only when the model views data, with a given classification task.
Intrinsic Dimension. Ansuini et al. (2019) measure the intrinsic dimension of layer representations for many common neural network architecture. At the same time, they measure the intrinsic dimension of MNIST, CIFAR-10 (Krizhevsky et al., 2010) and ImageNet (Deng et al., 2009) datasets. Our objective is similar, but we do not specifically quantify the dimension of the data manifold. The data leaf reflects how a classifier sees a geometric structure on the discrete data points from a dataset. Its dimension is intimately linked to the classification task. For this reason the results are not directly comparable.
Ansuini et al. (2019) show that MNIST and neural networks trained on it behave very differently from networks trained on CIFAR-10 or ImageNet. While our experiments focus on MNIST, additional experiments on CIFAR-10 are shown in the Appendix C.
Adversarial Attacks. Our method to navigate the leaves of the foliation is very similar to common adversarial attack methods, like Fast Gradient Sign Method (Goodfellow et al., 2015) or Projected Gradient Descent. Adversarial attacks and our navigation algorithm both rely on gradients∇x log pi, but adversarial generation algorithms perturb the original image by sign(∇x log pi). In general, sign(∇x log pi) /∈ ker(G(x,w))⊥, so adversarial examples are created perturbing the image outside the data leaf.

heading: 6 CONCLUSIONS
text: In this paper, we introduce the local data matrix, a novel mathematical object that sheds light on the internal working of a neural network classifier. We prove that the model organizes the data domain according to the geometric structure of a foliation. Experiments show that valid data are placed on the same leaf of the foliation, thus the model sees the data on a low-dimensional submanifold of the data domain. Such submanifold appears more general than the model itself, because it includes meaningless, but visually similar, images together with training and test data.
In the future, we aim to characterize the data leaf and to study the Riemannian metric given by the local data matrix. If we could analytically characterize the leaves of the foliation by the degree of noise, we could distinguish noisy data from valid examples. That can be used in the inference phase to exclude examples on which model predictions are not reliable. Furthermore, it could pave the way to the creation of novel generic denoising algorithms applicable to every kind of data.

heading: A APPENDIX: FROBENIUS THEOREM
text: For the reader’s convenience, we collect here few facts regarding differentiable manifolds, for more details see Abraham et al. (2012), Tu (2008).
Let M be a differentiable manifold. Its tangent space TxM at a point x ∈ M can be effectively defined as the vector space of derivations at x, so TxM = { ∑ i a
i∂i}, after we choose a chart around x. A vector field X on M is a function x 7→ Xx ∈ TxM assigning to any point of x ∈ M a tangent vector Xx. An integral curve of the vector field X at p is a map c : (− , ) −→ M , such that c(0) = p and ddtc(t) = Xc(t), for all t ∈ (− , ), ∈ R.
For example, in R2, we can define the vector field: X : (x, y) 7→ −y∂x + x∂y
One may also view X as assigning to a point p = (x, y) in R2 the vector (−y, x) applied at p. The integral curve of X at (1, 0) is the circle c(t) = (cos(t), sin(t)), t ∈ (−π, π), in fact c′(t) = (−sin(t), cos(t)) = Xc(t). More in general, the integral curve of X at a generic point p is the circle centered at the origin and passing through p. If TM = ∐ x∈M TxM (disjoint union) denotes the
tangent bundle, we also call the vector field X a section of TM .
Since TxM is the vector space of derivations on differentiable functions defined on a neighbourhood of x, we can write Xx(f); this is the result of the application of the derivation Xx ∈ TxM to the function f ∈ C∞M (U), where C∞M (U) denotes the differentiable functions f : U −→ R, x ∈ U open in M . As x varies in U , Xx(f) defines a function, which we denote with X(f).
We call χ(M) the smooth vector fields on M , that is, those vector fields expressed in local coordinates as Xx = ∑ i ai(x)∂i, with ai : U −→ R smooth functions, x ∈ U , U open in M .
We can define a bracket on the space of vector fields as follows: [X,Y ](f) := X(Y (f))− Y (X(f)), f ∈ C∞M (M), X, Y ∈ χ(M)
[ , ] is bilinear and satisfies: 1. [X,Y ] = −[Y,X] (antisymmetry); 2. [X, [Y,Z]] + [Y, [Z,X]] + [Z, [X,Y ]] = 0 (Jacoby identity).
We are ready to define distributions, which play a key role into our treatment. Let M be a differentiable manifold. We define a distribution D of rank k an assignment:
x −→ Dx ⊂ TxM, ∀x ∈M where Dx is a subspace of TxM of dimension k. We say that D is smooth if at each point x ∈ M , there exists an open set U , x ∈ U , Dy is spanned by smooth vector fields on U for all y ∈ U . We say that a distribution is integrable if for any x ∈ M there exists a local submanifold N ⊂ U , U open in M , x ∈ N , called a local integrable manifold such that TzN = Dz for all z ∈ N . When such N exists globally, we say we have a foliation of M , that is we can write M as the disjoint union of integrable submanifolds, all of the same dimension and immersed in M . Each integrable submanifold of the foliation is called a leaf. For example in R3 \ {(0, 0, 0)}, the distribution:
p = (x, y, z) 7→ Dp = (span{x∂x + y∂y + z∂z})⊥ (11)
is integrable: at each point p, Dp is the tangent plane to a sphere centered at the origin and passing through p. Hence we can write the space R3\{(0, 0, 0)} as the disjoint union of spheres: each sphere is a leaf of the foliation thus obtained. We may also say that R3 \ {(0, 0, 0)} foliates as the disjoint union of spheres. Another example is given in Figure 7, showing how the distribution generated by the vector field X foliates R2 \ {(0, 0)} as the disjoint union of circles. We say that a distribution is involutive if for all vector fields X,Y ∈ D we have [X,Y ] ∈ D. Frobenius Theorem establishes an equivalence between these two key properties of a distribution, namely integrability and involutivity, thus giving us an effective method to establish when a distribution gives a foliation of M into disjoint submanifolds. Theorem A.1. Frobenius Theorem LetM be a differentiable manifold andD a distribution onM . Then D is involutive if and only if it is integrable. If this occurs, then there exists a foliation on M , whose leaves are given by the integrable submanifolds of D.
Proof. See Abraham et al. (2012) Section 4.4.
Frobenius Theorem tells us that an involutive distribution onM defines at each point p a submanifold N of dimension k of M . This means that, locally, we can choose coordinates (x1, . . . , xn) for M , so that, in the chart neighbourhood of p, the submanifold N is determined by equations xk+1 = ck+1, . . . , xn = cn, where ci are constants. We have then that (x1, . . . , xk) are local coordinates for the submanifold N around p. Notice that the vector fields associated to these particular coordinates Xi = ∂xi verify a stronger condition than involutivity, namely [Xi, Xj ] = 0. Furthermore, the proof of Frobenius theorem is constructive: it will give us the vector fields Xi and the coordinates xi as their integral curves. Let us see in a significant example, what this construction amounts to.
We express the distribution D defined in (11) explicitly as:
p = (x, y, z) 7→ Dp = (span{x∂x + y∂y + z∂z})⊥ =
= span{X = y∂x − x∂y, Y = z∂x − x∂z} (12)
As one can check [X,Y ]p = −y∂z + z∂y ∈ Dp for all p ∈ R3 \ {(0, 0, 0)}. With Gauss reduction we transform (
y −x 0 z 0 −x
) −→ ( 1 0 −x/z 0 1 −y/z ) One can then verify that [X1, X2] = 0 for X1 = ∂x − (x/z)∂z , X2 = ∂y − (y/z)∂z . The calculations of this example are indeed a prototype for the treatment in the general setting: given a local basis for an involutive D, that is a set of vector fields X1, . . . Xk, which are linearly independent in a chart neighbourhood, using Gauss algorithm, we can transform them into another basis Y1 . . . Yk with the property [Yi, Yj ] = 0. The integral curves of the Yi’s will then give us the coordinates for the leaf manifold in the given chart.

heading: B APPENDIX: PROOFS
text: In this appendix we collect the proof of the mathematical results stated in our paper. Recall the two key definitions of local Fisher and data matrices:
F (x,w) = Ey∼p[∇w(log(p(y|x,w)) · ∇w(log(p(y|x,w))T ] (13) G(x,w) = Ey∼p[∇x(log(p(y|x,w)) · ∇x(log(p(y|x,w))T ] (14)
We start with the statement and proof of Proposition 2.1. Proposition B.1. Let the notation be as in Section 2. Then:
1. kerF (x,w) = span{∇w log pi(y|x,w)|1 ≤ i ≤ C}⊥; kerG(x,w) = span{∇x log pi(y|x,w)|1 ≤ i ≤ C}⊥.
2. rank F (x,w) < C, rank G(x,w) < C.
Proof. We prove the results for F (x,w), the proofs for G(x,w) are akin. 1. We show that kerF (x,w) ⊆ span{∇w log pi(y|x,w)|1 ≤ i ≤ C}⊥:
u ∈ kerF (x,w)⇒ uTF (x,w)u = 0⇒ Ey∼p [ 〈∇w log p(y|x), u〉2 ] = 0
⇒ 〈∇w log pi(y|x), u〉 = 0 ∀ i = 1, . . . , C. (15)
On the other hand, if u ∈ span{∇w log pi(y|x,w)|1 ≤ i ≤ C}⊥ then u ∈ kerF (x,w): F (x,w)u = Ey∼p [∇w log pi(y|x)〈∇w log pi(y|x), u〉] = 0. (16)
2. From 2.1.1, we know that rank F (x,w) ≤ C. Equation (4) allows us to conclude the proof.
We now go to the main mathematical result of our paper, which is the key to provide with a manifold structure the space of data. Theorem B.2. Let w be the weights of a deep ReLU neural network classifier, p given by softmax, G(x,w) the local data matrix. Assume G(x,w) has constant rank. Then, there exists a local submanifold N ⊂ Rn, x ∈ N , such that its tangent space at z, TzN = ker(G(z, w))⊥ for all z ∈ N .
Proof. By Frobenius Theorem, we need to check the involutivity property (10) for the distribution x 7→ Dx = ker(G(x,w))⊥ on Rn. Since by Prop. 2.1,
Dx = ker(G(x,w))⊥ = span{∇x log pk(y|x,w) | 1 ≤ k ≤ C} we only need to show that:
[∇x log pi(y|x,w),∇x log pj(y|x,w)] ∈ span{∇x log pk(y|x,w) | 1 ≤ k ≤ C} By standard computations, we see that:
[∇x log pi(y|x,w),∇x log pj(y|x,w)] = H(log pi(y|x,w))∇x log pj(y|x,w)+
−H(log pj(y|x,w))∇x log pi(y|x,w), (17)
where H(f) denotes the Hessian of a function f . Here we are using the fact that [ ∑ i a∂i, ∑ j bj∂j ] = ∑ i,j (ai∂ibj − bi∂iaj)∂j
With some calculations, we have:
H(log pi(y|x,w)) = H(pi(y|x,w)) pi(y|x,w) −∇x log pi(y|x,w) · (∇x log pi(y|x,w))T (18)
and
∇xpi(y|x,w) = C∑
k=1
pi(y|x,w)(δik − pk(y|x,w))∇xsk, (19)
where s is the score function and we make use of the fact pi(y|x,w) is given by softmax. Notice ∂skpi(y|x,w) = pi(y|x,w)(δik − pk(y|x,w)), where δik is the Kronecker delta. By (18) and (19):
H(pi(y|x,w)) = Jac(∇xpi(y|x,w)) = C∑
k=1
∇x [pi(y|x,w)(δik − pk(y|x,w))] (∇xsk)T , (20)
where we use the fact that sk is piecewise linear so that its Hessian is zero, where it is defined. Hence:
H(pi(y|x,w)) = ∇xpi(y|x,w)(∇xpi(y|x,w))T
pi(y|x,w) − pi C∑ k=1 ∇xpk(y|x,w)(∇xsk)T . (21)
Now, in view of (17), using (18) and (21) we compute the expression:
H(log pi(y|x,w)) = ∇xpi(y|x,w) pi(y|x,w) ( ∇xpi(y|x,w) pi(y|x,w) )T − C∑ k=1 ∇xpk(y|x,w)(∇xsk)T +
−∇x log pi(y|x,w) · (∇x log pi(y|x,w))T = − C∑
k=1
∇xpk(y|x,w)(∇xsk)T .
(22)
Hence:
H(log pi(y|x,w))∇x log pj(y|x,w) = − C∑
k=1
∇xpk(y|x,w)(∇xsk)T∇x log pj(y|x,w).
Since (∇xsk)T∇x log pj(y|x,w) is a scalar, we obtain H(log pi(y|x,w))∇x log pj(y|x,w) is a linear combination of vectors∇xpk(y|x,w), and therefore a linear combination of∇x log pk(y|x,w). The same holds for [∇x log pi(y|x,w),∇x log pj(y|x,w)] as we wanted to show.

heading: C APPENDIX: CIFAR-10 EXPERIMENTS
text: We perform some experiments with the CIFAR-10 dataset and a neural network similar to VGG-11.

Review summaries:
The authors propose a so called data matrix that is induced in the input space of a deep neural network classifier. This matrix is similar to the Fisher-Rao metric, but for the input and not the parameters of the model. The analysis of this matrix shows that the classifier induces in the input space a specific structure, which the authors study. Constructive experiments are used in order to empirically verify the claims.

In this work, the authors showed that deep ReLU networks can model the low dimensional manifold structure of the dataset. The authors first define a local data matrix G which is analogous to Fisher matrix. Then they proved that the tangent space of the data manifold is spanned by the eigen vectors of G corresponding to non-zero eigen values. The authors visualize this data manifold on MNIST data.