Model-centric data manifold: the data through the eyes of the model
Authors: Luca Grementieri
Rita Fioresi
Venue:  (2021)
Keywords: Deep Learning
Information Geometry
Data Manifold
Fisher matrix

Abstract: We discover that deep ReLU neural network classifiers can see a low-dimensional Riemannian manifold structure on data. Such structure comes via the local data matrix, a variation of the Fisher information matrix, where the role of the model parameters is taken by the data variables. We obtain a foliation of the data domain and we show that the dataset on which the model is trained lies on a leaf, the data leaf, whose dimension is bounded by the number of classification labels. We validate our results with some experiments with the MNIST dataset: paths on the data leaf connect valid images, while other leaves cover noisy images.


Under review as a conference paper at ICLR 2021
MODEL-CENTRIC DATA MANIFOLD: THE DATA THROUGH THE EYES OF THE MODEL
Anonymous authors Paper under double-blind review
ABSTRACT
We discover that deep ReLU neural network classifiers can low-dimensional Riemannian manifold structure on data. Such structure comes via the local data matrix, a variation of the Fisher information matrix, where the role of the model parameters is taken by the data variables. We obtain a foliation of the data domain and we show that the dataset on which the model is trained lies on a leaf, the data leaf, whose dimension is bounded by the number of classification labels. We validate our results with some experiments with the MNIST dataset: paths on the data leaf connect valid images, while other leaves cover noisy images.
INTRODUCTION
In machine learning, models are categorized discriminative models or generative models. From its inception, deep learning has focused on classification and discriminative models (Krizhevsky et al.]
2012 Hinton et al. 2012 Collobert al. (011)) Another perspective came with the construction of generative models Is based on neural networks (Kingma & Welling, 2014 Goodfellow et al. 2014]
Van den Oord et al.) 2016;; Kingma Dhariwal (0118). Both kinds of models give us information about the data and the similarity between examples. In particular, generative models introduce a geometric structure on generated data. Such models transform random low-dimensional vector to an example sampled from probability distribution approximating the one of the training dataset. As proved by Arjovsky Bottou (2017), generated data lie on a countable union of manifolds. This fact supports the human intuition that data have low-dimensional manifold structure, but in generative models the dimension of such manifold usually a recent algorithm by Peebles et al. (2020) provides way find an approximation of the number of dimensions of the data manifold, deactivating irrelevant dimensions a GAN.
fixed by the experimenter. A
Similarly, here we try to understand if discriminative model can be used to detect manifold structure on the space containing data and to provide tools to navigate this manifold. The implicit definition of such manifold and the possibility to trace paths between points on the manifold can open many possible applications. In particular, we could use paths to define a system of coordinates on the manifold (more specifically on chart of the manifold). Such coordinates would immediately give us a fow-dimensione parametrization of our data, allowing us to do dimensionality reduction.
In supervised learning, model trained on a labeled dataset identify the correct label on unseen data. A trained neural network classifier builds a hierarchy of representations that encodes increas- ingly complex features of the input data (Olah al. 2017). Through the representation function, distance (e.g. euclidean or cosine) on the representation space of a layer endows input data with a distance. This pyramid of distances on examples increasingly class-aware: the deeper is the layer, the better the metric reflects the similarity of data according to the task at hand. This observation suggests that the model implicitly organizing the data according to suitable structure.
Unfortunately, these intermediate representations and metrics are insufficient to understand the geo- metric structure of data. First of all, representation functions are not invertible, so we cannot recover the original example from its intermediate representation or interpolate between data points. More- over, the domain of representation functions is the entire data domain R". This domain is mostly composed of meaningless noise and data occupy only thin region inside of it. So, even if rep- resentation functions provide us a distance, those metrics are incapable of distinguishing between meaningful data and noise.

---

Under review as a conference paper at ICLR 2021
We find out that ReLU neural network implicitly identifies a low-dimensional submanifold of the data domain that contains real data. We prove that if the activation function is piecewise- -ineerr--eerrrrree (e.g. ReLU), the neural network decomposes the data domain I" as the disjoint union of submanifolds (the leaves of a foliation, using the terminology of differential geometry). The dimension of every submanifold (every leaf of the foliation) is bounded by the number of classes of classification model, so it is much smaller than n, the dimension of the data domain R".. Our main theoretical result, Theorem 3.1 stems from the study of the properties of variant of the Fisher Information matrix, the local data matrix. However, Theorem 3.1 cannot tell us which leaves of this foliation are meaningful, i.e. what are the possible interesting practical applications of the submanifolds that compose the foliation. The interpretation of this geometric structure can only come from ex- periments. We report experiments performed on MNIST dataset. We choose to focus on MNIST because it is easily interpretable and because small networks are sufficient to reach high accuracy.
Our experiments suggest that all valid data points lie only one leaf of the foliation, the data leaf. To observe this phenomenon we take an example from the dataset and we try to connect it with other random example following a path along the leaf containing the starting point. If such path exists, it means that the destination example be- longs the same leaf of the foliation. Visualizing the intermediate points on these joining paths, we see that the low-dimensional data manifold defined by the model is not the anthropocentric data man- ifold composed of data meaningful for a human observer. The model-centric data manifold com- prises images that do not belong to a precise class. The model needs those transition points to connect points with different labels. At the same time, it understands that such transition points represent an ambiguous digit: on such points, the model assigns low probability every class.
2
noise leaf
data leaf
S
S R 2
2 2
Figure I:
Simplified summary of our experi- ments: images from MNIST are connected by paths the data Icaf, while images on other leaves are noisy.
The experiments also show that moving orthogonally to the data leaf we find noisy images. That means that the other leaves of the foliation contain images with level of noise that increases with the distance from the data leaf. These noisy images become soon meaningless to the human eye, while the model still classifies them with high confidence. This fact is a consequence of the property of the local data matrix: equation
(8) prescribes that the model output does not change if we move in direction orthogonal to the tangent space of the leaf on which our data is located.
This remark points us to other possible applications of the model-centric data manifold. We could project a noisy point on the data leaf perform denoising, or we can use the distance from the data leaf to recognize dittiiiibbttooonnn examples.
The main contributions of the paper are: 1. the definition of the local data matrix G(ar, w) at point x of the data domain and for given model w, and the study of its properties;
2. the proof that the subspace spanned by the eigenvectors with non-zero eigenvalue of the local data matrix G(æ, w) can be interpreted as the tangent space of a Riemannian manifold, whose dimension is bounded by the number of classes which our model is trained;
3. the identification and visualization of the model-centric data manifold through paths, obtained via experiments MNIST.
Organization of the paper. In Section 2 we review the fundamentals of information geometry using novel perspective that aims at facilitating the comprehension of the key concepts of the paper. We introduce the local data matrix G(z, w) and we summarize its properties in Prop. 2. In Section 3] we show that, through the local data matrix, under some mild hypotheses. the data domain foliates as a disjoint union of leaves, which are all Riemannian submanifolds of IR", with metric given via G(z w). In Section 4 we provide evidence that all our dataset lies on one leaf of the foliation and that moving along directions orthogonal to the data leaf amounts to adding noise to data.
2

---

Under review as a conference paper at ICLR 2021
2 INFORMATION GEOMETRY
Here we collect some results pertaining to information geometry (Amari 1998; Nielsen 2018 using novel perspective adapted to question, namely how provide manilold structure to the space containing data.
Let p(ylx, w) be discrete probability distribution on C classification labels, i.. p(ylæ, w) (pi (yla, [r, w))i=1,..... 2 E Σ C IR", w E RR. the applications, I represent input data belonging to certain dataset Σ, while are the learning parameters, i.e. the parameters of the empirical model. As we are going to see in our discussion later on, it is fruitful to treat the two sets of variables x and on equal grounds. This will naturally lead geometric structure on a low dimensional submanifold of IR", that we can navigate through paths joining points in the dataset 2 (see Section
In order to give some context to our treatment, we define, following Amari 1998) information loss [xx w) log(p(y|ær, w)) and the loss function Lxx, (r, w) Eynug I (z, w)). Typ- ically L(x, w) is used for practical optimizations, where we need to compare the model output distribution p(y|æ, w) with certain known true distribution g(y ar). We may also view [[x, w) as the Kullback-Leibles divergence up to the constant optimization problem:
Section 3, the
>. (v(22) log (v(z)) irrelevant for any
(lyl)) ((yl(r) pi(u|æ, w)
L(x, w) = Ey-ul- log(p(yl w)]] =
(1lk)) log (x(xz)
-Ku.(glll=)lllolle w))
Σ log
popular choice for (yyla, w) in deep learning classification algorithms is
(a(,w))) (*:(2,,))
(ul 1r w) = softmax (I, sc(a, w));
(2)
j=ml
where s(x, w) E RC is a score function determined by parameters w. From such plylæ, w) we derive the cross-entropy with softmax loss function:
Mxx w) EEy~gl w)] Eny=g log plylo, w)) -8y, (as, w) + log
(x,w)
where L(x, w) is computed with respect to the probability mass distribution ((y|2)) assigning 1 to the correct label Yr of our datum x and zero otherwise. Other approaches rely on label smoothing (Szegedy al., 2016)), hence they take different L(x, w). However, since our treatment mainly relies on the expression of ((r, w). our results will also apply to such loss functions, provided some hypotheses, that we list Section 3 are satisfied.
Going back to the general setting, notice that: EyypI V we (I,, w)] = 0. In fact,
Ey-p V (log(pl)|z, w))l
Σ lo
Vu v =
(4)
Let us now define the following two matrices:
F(x, w) = Ey-p V (log(p(vlz w)) V (log(p(ulz, w)))
(5)
G(æ, w) Ey-pi V (log(p(ylz, w)) V (log((((y)z,, w))²
(6)
We call F(x, w) the local Fisher matrix at the datum æ and G(æ,, w) the local data matrix given the model The Fisher matrix (Amari 1998) is obtained gives information on the metric structure of the space of parameters. Similarly, we can reverse our perspective and see how G(z, w), allows to recognize
F(w) EF=x[F(2, w)] and it
structure in dataset.
The following observations apply to both F(x, w) and G(x, w) and provide the theoretical corner- stone of Section 3
3

---

Under review as a conference paper at ICLR 2021
Proposition 2.1. Let the notation be as above. Then:
1. ker F(x,, w) span V (og.: w) < i + ker G(r,, w) = span V2 logpity w sis C
2. rank F(x, w) < C,
rank G(z,w) <C
Proof. See Appendix
B
This result tells us that the rank of both F(x, w) and G(z, w) is bounded by C, the number of classes in our classification problem. The consequence of the bound on rank F(z, w) the following: in SGD dynamics with single example, the number of directions in which the change in our param- eters modifies our loss severely limited. The consequence on the bound on rank G(x, w) even more striking: it will allow to define submanifold of Rº" of dimension rank G(z, w)) (Section that our experiments show contains our dataset (Section 4 In practical situations, this dimension is much lower than the size of G( w), i.e. the input size as shown in Table I
Table 1: Bound on the rank of Gax, w) for popular image classification tasks.
Dataset MNIST
Size G(z, w)
Bound on rank G(x, w)
784
10
CIFAR-10
3072
10
CIFAR-100
3072
100
ImageNet
150528
1000
THE MODEL VIEW ON THE DATA MANIFOLD
We now turn to examine some properties of the matrices F(x, w) and G(x, w) that will enable us to discover a submanifold structure on the portion of Re occupied by our dataset (see Section 4 and to prove the claims at the end of the above section.
We recall that, given a perturbation of the weights w, the Kullhack-Leibler divergence, gives, to second-order approximation, the following formula:
(KL(p(ulz, 5w)llp() w)) = (Rw)² F(x, ()( + 0(1680r
Equation together with Proposition 2.1, effectively expresses the fact that during SGD dynamics with a mini-batch of size 1, we have only a very limited number of directions, namely C 1, in which the change dw affects the loss.
Taking the expectation with respect to x ~ E on both sides of the equation, we obtain the analogous property for the Fisher matrix F(w)
Em>>[f(x w)]. While F(x, has low rank, the rank of F(x) is bounded by [E] 1). number that often higher than the size of F(x). Thus F(w). when non-degenerate, is an effective metric on the parameter space. It allows to measure, according to a certain step ōw, when we reach a stable predicted probability and thus the end of model training (see Martens 2020) and refs. within).
must be however noted that F(w) retains its information content only away from the trained model, that is, well before the end of the training phase (see AAhille et al. (2018) for empirical validation of such statements). We are going to see, with our experiments in Section 4 that a similar phenomenon occurs for G( w). Our Fig. elucidates the rank behaviour of G(x, w) during training.
We now turn to the local data matrix G(æ,, w). thus interpreting equation 7 in the data domain R For perturbation åa of the data H, we have, up to second order approximation:
DKL. (pc.(p(y/ar oo + Jx. w)llpp ar w)) (6x)²² G(v,, ()x))) 0(1622)
(8)
4

---

Under review as a conference paper at ICLR 2021
This equation suggests view G(z, w) as metric the data domain. However, because of its low rank (see Proposition 2.1). we need to restrict our attention to the subspace ker(G(a, w)) where G(x, w) is non-degenerate. G(x, w) allows us to define distribution D on R". In general, in differential geometry, we call distribution on Rn an assignment:
I - D2 C IR""
Vx R"
where Dr is a vector subspace of R of fixed dimension & (see Appendix A for more details on this notion in the general context).
Assume now Gz w) has constant rank; as we shall see in our experiments, this is the case for non fully trained model (see Section A more details). We thus obtain distribution D:
r - D= = ker(G w))² E
Equation (8) tells us that, if we move along the directions of ker (G((, w)), the probability distribu- tion p(y|2, 2, w) is constant (up to a second order approximation) while our data changing. Those are the vast majority of the directions, since rank Giæz, w) < C and typically C' << n, hence, we interpret them as the noise directions. On the other hand, if we move from data point x along the directions in ker(G( w))- data will change along with the probability distribution p(y (r, w) associated with it. These are the directions in which the model moves with confidence; we are going to clarify this key point in Section
We now would like to see if our distribution 4 defines a foliation structure. This means that we can decompose In as the disjoint union of submanifolds, called leaves of the foliation, and there a unique submanifold (leaf) going through each point x (see Fig. 7 in Appendix A where the distribution is generated by the vector field X and R2 is the disjoint union of circles, the leaves of the foliation). The distribution comes into the play, because it gives the tangent space to the leaf through æ: D, ker(G(( x²)) In this way, moving along the directions in D2 each point x, will produce a path lying in one of the submanifolds (leaves) of the foliation.
The existence of foliation, whose leaf through a point has tangent space Da,, comes through Frobenius theorem, which we state in the Appendix A and we recall here in the version that we need.
Frobenius Theorem. Let a E Rn and let DD be a distribution in R", Assume that in a neighbourhood U of
E Then, there exists (local) submanifold N C R", x E N, such that T.N DD. all z N.
AX,Y
(10)
It is not reasonable to expect that a general classifier satisfies the involutive property (10). however it is remarkable that for a large class of classifiers, namely deep ReLU neural networks, this is the case, with p given by softmax as in equation (2)
Theorem 3.1. Let w be the weights of deep ReLU neural network classifier, p given by softmax, G(x, w) the local data matrix. Assume G(x, w) has constant rank. Then, there exists a local sub- manifold N C R E N, such that its tangent space at 2, T.N = =krr(G(z, ()) for all 2 E N.
Proof. See Appendix B
Through the application of Frobenius theorem, Theorem 3.1 gives us a foliation of the data domain R". R" decomposes into the disjoint union of C dimensional submanifolds, whose tangent space at a fixed x R is D ker(G(z, w)) (see Appendix AA.. Every point x determines a unique submanifold corresponding to the leaf of the foliation through x. We may extend this local submanifold structure to obtain global structure of manifold on leaf, still retaining the above property regarding the distribution.
As we shall see in Section 4 we can move from a point x in our dataset Σ to another point x' also in Σ with an horizontal path, that is path tangent to Dz. hence lying on the leaf of x and x'. Our experiments show that we can connect every pair of points (z, 2') E Σ x Σ with horizontal paths. means that all the dataset belongs to single leaf, which we call the data leaf L. Our model, through the local data matrix Gz w), enables us to move on the now-dimensional submanifold E, to which all of our dataset belongs. Of course not all the points of L correspond elements of the dataset;
5

---

Under review as a conference paper at ICLR 2021
however show in the experiments, most points of L the model gives prediction compatible with human observers.
We also notice that each leaf of our foliation comes naturally equipped with metric given each point by the matrix G(z, w), restricted to the subspace ker ker(G((r, w))+, which coincides with the tangent space to the leaf, where G(æ,, w) is non-degenerate. Hence G(r,, w) will provide each leaf with a natural Riemannian manifold structure. We end this section with an observation, comparing our approach the geometry of the data domain, with the parameter space.
Remark. Equation 7 provides a metric to the parameter space R4, motivating our approach to the data domain. For each w. + D' == ker ker(F(w)) (F(w))++ using the Fisher matrix. However, it is easy to see empirically that this distribution is not involutive, i.c. there no foliation and no submanifold corresponding to it.
IR,
define, as did for the data domain, distribution
EXPERIMENTS
We performed experiments on the MNIST dataset (LeCun al. 1998) and on the CIFAR-10 dataset (Krihhessyy al. 2010). We report in this section the experiments on the MNIST dataset only, because they are easier to interpret in the geometrical framework (foliation and leaves) introduced in our previous section. CIFAR- 10 experiments reported in the dedicated Appendix C Thus, all the following experiments use a simple CNN classifier trained on the MNIST.
The definition of distribution in Section 3 and the consequent results require the rank of G( w) to be constant on every point x for a certain parameter configuration w. We remark that the rank of G(r,, w) can be most C' (Proposition i.e. much lower than the size of G(x, w) (Table [ so we can expect that rank G(x, w) = C for almost all x E IR.
We observe that during the training of the CNN, the rank of G(x, w) is stable and equal to C - 1. That changes at the end of the training, when the model reaches convergence. At that point, the output class probability distribution on most images highly unbalanced and sparse and all the eigenval- ues of G(x, w) tend to O. Figure 2 shows the decreasing trend of the mean trace of G(r, w) on batch during the training. The plot is similar to the ones reported in Achille et al. (2018) for the Fisher information matrix and it tells us that the local data matrix Gzz, w) loses its informative content the end of training as well. This observation motivates us to perform subsequent experiments
Figure 2: Mean trace of G(I,, w) during training
partially trained model and with data points where every output class probability is lower than 0.99.
Since a complex model reaches convergence on MNIST digit classification too fast, we have resorted to a very small CNN. The network used in the following experiments has 18,982 parameters and ReLU activation function. is similar LeNet, but with channels in both the convolutional layers and 32 hidden units in the fully-connected layer. We train this network with SGD for just one epoch to prevent model convergence.
Such a neural network classifier satisfies all the hypothesis of Theorem 3.1 So it views the data domain R as foliation. However, this result does not give us any clue about the characterization of leaves. We know, however, that, moving away from a data point z while remaining on the same leaf, we can obtain points with different labels, while moving in a direction orthogonal to the tangent space to the leaf of v, we obtain points with the same label as as long as the estimate ( holds.
To understand the distinguishing factors of a leaf, we move from a data point x across leaves and inspect the evolution. We start from image in MNIST test set and we let it evolve moving orthogonally to leaves. In order to get further and further away, we use a fixed random direction and the space orthogonal the current leaf. From the previous section, know that such space coincides with ker G(r,w), w).
every step we project
We observe in Figure that the noise in the images increases steadily. Moreover, can that, as predicted by equation (8), model predictions remain very certain even when the digit is indistinguishable for the human eye.

---

Under review as a conference paper at ICLR 2021
preicted hecel 2 with predicted lase v th predics Weh wit predicted lhe with peed cne Ahe wm powd wwwerree labal 2 AP pwweiti 16S4
2
FOOkEeN NEW Sith preditted INs whth piddtie Hne WIT prwseted Nh ait pod (TE ANW aalm prodiceed Inbal WPt poodined lasel Cath piectted label unn
SAS
5
5
5
Figure 3: Paths across leaves of the foliation from valid image.
This experiment makes speculate that leaf is characterized by constant amount of noise. If that is the case, all valid data should reside on the same leaf characterized by the absence of noise, the data leaf. To verify our intuition we follow horizontal paths connecting two images from MNIST test set. If it is possible to join two inputs with a horizontal path, then we know that those points are on the same leaf. In our case horizontal path is tangent to the distribution D described in the previous section, i.e. Dr (ker G(x, w²)) span V. log Ps (ylar, w)[1 S i S C} from Proposition We remark that Dr coincides with the row space of the Jacobian matrix Jacz p(x, w). We will use this equivalence to efficiently compute the projection of a vector on the distribution Dz.
The algorithm to approximate path from a source point to a destination point is a simplification of Riemannian gradient descent (Gabay) 1982) using the euclidean distance from the destination point as loss function. Since we do not have an explicit characterization of the leaf, we cannot perform the retraction step commonly used in optimization algorithms manifold. To circumvent this problem, we normalize the gradient vector and use a small step size of 0.1 in our experiments. The normalization assures that the norm of the displacement at every step is controlled by the step size. The iterative algorithm is reported in Algorithm [
Algorithm 1: Find horizontal path between source and destination points.
Input: source; d: destination; step size, T: number of iterations; model parameters.
To < 5;
for f 1, 1...., do
j + Jac, log p(x,-1(111 1, w) (Calculate Jacobian of log p(#,-1, w) w.r.t. xz-1);
v + projection(d (d x2- (Project the gradient of ||d 1221 2 on D Dz._1);
St f- 14-1 + 0 |||| (Update 2 to obtain the next point I3 on the horizontal path); return (
Derstlon preskdal bazi with presidedd laas prodkded lubd with prossted Lidd prod eled with
Beration 1292:
Seration 2500
Tenatin 3750
Beneicn Jeralian EDSC: Derratonn 7502: Feretion E750:
Beeraion 1003C:
prokeczd label wét prodited label 2 prosided lubd with uud
2
2
2
2
Derstion preskded hazzl with p1 by 30:33
Deration
Peratinn 2500
haration 3150
Bennicn Inratian 63SC:
Derstion 7593:
berston 8750: Benalon 1903:
presided laos a th probuiiilyy E338
prodided lube wiit incebiilly 6.607
prossted lbd with prod abe with
prodectdd label with
predited lasel Bath procided labe with
grsteteily onstcteity
poosstedy E3776
prosa14ly prpd0 e
8
8
8
Figure 4: Horizontal paths between two images in MNIST test set. Here and all the following images the pairs of source-destination points are sampled randomly from the test set.
7

---

Under review as a conference paper at ICLR 2021
Figure 4 shows the results of the application of Algorithm on some pairs of images in MNIST test set. We observe that is possible to link very different images with an horizontal path, confirming conjecture about the existence of the data leaf. This experiment shows that the model sees the data on the same manifold, namely one leaf of the foliation determined by our distribution D.
The observation of the paths the data leaf gives property of the model. First of all, while we could expect to find all training data on the same leaf, is remarkable that test data are placed on the same leaf too. Furthermore, leaf is not limited to digits and transition points between them. In fact, we can find paths connecting valid images with images of non-exte non-existent digits similar letters, as shown in Figure That fact suggests that the model-centris data manifold is somewhat more general than the classification task on which the model is trained.
novel point of view the generalization
observe that the data
(teration prbba is 201
Beraltnn 1252: last provesillyy No37
Bereton 2300 libd pribeniity 5413
3150
5000
leratian
Deriion 7502: lasel prssii 3974
barsiin E7S0:
1923C: pros kddd ziitadi 10w
proceeeiry 1515
ano onttederty $5:50
prsssts By 9677
presin byy 5107
Figure 5: Horizontal path between mirrored image and valid image from MNIST test set.
Our final experiment gives us another remarkable confirmation of our theoretical findings: it is impossible link with horizontal path a noisy image outside the data leaf and a valid data image. Figure 6 shows that even after thousands of iterations of Algorithm I it is impossible to converge to the destination point. Indeed, the noise preserved along the path and the noise pattern stabilizes after 5000 iterations. The final point reached is noisy version of the actual destination point.
prokited lac!l with precktte load with unoiktiall lued will
procted Lide
srol sted oldd Giit produce lile 1 with prokiial latel with stodkloa Laad proluted uddd wen
presktted labl with preskted las I anoiktial ubd will prosted Lid wis pro sied dldd 2 Fiih prokeco lobel with prosiicd lubel 5'tth procstel Udd with prodittd Mdd wi
rybabb
3
3
3
3
3
3
Figure 6: Horizontal paths unable reach a valid image in MNIST test from a noisy image.
All those experiments confirm that there exists model-centric data manifold, the data leaf of the foliation. In addition they exhibit that the leaves in the foliation of the data domain are characterized by the noise content.
5 RELATED WORKS
Fisher Information Matrix. In Achille et al. (2018) the authors discuss the information content of the Fisher matrix, and they show that such content changes during the training of neural network, decreasing rapidly towards the end of it. This shows that the Risher-Rao metric acquires importance during the training phase only (see also Kirkpatrick et al. (2017)). More on the geometry of the parameter space, as a Riemannian manifold, is found in Sommer & Bronstein (202)). In this paper, we take an analog of the Fisher-Rao metric, but on the data domain. In our examples, we see a phenomenon similar to the observed in Achille et al. (2018): the trace of the local data matrix decreases rapidly, as the model completes its training. Other metrics on datasets were suggested, for example see Dukler et al. (201)) and refs. within, but with different purposes. Here, our philosophy
8

---

Under review as a conference paper at ICLR 2021
is the same as in Bergomi et al. (2019): we believe that data itself is not equipped with a geometric structure, but such structure emerges only when the model views data, with a given classification
task.
Intrinsic Dimension. Ansuini et al. (019) measure the intrinsic dimension of layer representations for many common neural network architecture. At the same time, they measure the intrinsic dimen- sion of MNIST, CIFAR-10 10 Krizhevsky et al., 2010) and ImageNet (Deng et al. 2009) datasets. Our objective is similar, but we do not specifically quantify the dimension of the data manifold. The data leaf reflects how a classifier sees a geometric structure on the discrete data points from a dataset. Its dimension is intimately linked to the classification task. For this reason the results are not directly comparable.
Ansuini et (2119) show that MNIST and neural networks trained on it behave very differently from networks trained on CIFAR-10 10 or ImageNet. While our experiments focus on MNIST, addi- tional experiments on CIFAR-10 10 are shown the Appendix
Adversarial Attacks. Our method to navigate the leaves of the foliation is very similar to common adversarial attack methods, like Fast Gradient Sign Method (Goodfellow et al. (015)) or Projected Gradient Descent. Adversarial attacks and our navigation algorithm both rely on gradients V, log pi, but adversarial generation algorithms perturb the original image by sign( V. log general, sign( V, a log p.) & ker(G((z, =))+ adversarial examples are created perturbing the image outside the data leaf.
CONCLUSIONS
In this paper, we introduce the local data matrix, a novel mathematical object that sheds light on the internal working of a neural network classifier. We prove that the model organizes the data domain according to the geometric structure of foliation. Experiments show that valid data are placed on the same leaf of the foliation, thus the model sees the data on a low-dimensional submanifold of the data domain. Such submanifold appears more general than the model itself, because it includes meaningless, but visually similar, images together with training and test data.
In the future, we aim to characterize the data leaf and to study the Riemannian metric given by the local data matrix. If we could analytically characterize the leaves of the foliation by the degree of noise, we could distinguish noisy data from valid examples. That can be used in the inference phase to exclude examples on which model predictions are not reliable. Furthermore, could pave the way to the creation of novel generic denoising algorithms applicable to every kind of data.
REFERENCES
Ralph Abraham, Jerrold E Marsden, and Tudor Ratiu. Manifolds, tensor analysis, and applications, volume 75. Springer Science & Business Media, 2012.
Alessandro Achille, Matteo Rovere, and Stefano Soatto. Critical learning periods in deep networks. In International Conference on Learning Representations, 2018
Shun-Ichi Amari. Natural gradient works efficiently in learning. Neural computation, 10(2):251- 276, 1998.
Alessio Ansuini, Alessandro Laio, Jakob H Macke, and Davide Zoccolan. Intrinsic dimension of data representations in deep neural networks. In Advances in Neural Information Processing Systems, pp. 6111 1-6122, 2019.
Martín Arjovsky and Léon Bottou. Towards principled methods for training generative adversarial networks. In International Conference on Learning Representations, 2017.
Mattia G Bergomi, Patrizio Frosini, Daniela Giorgi, and Nicola Quercioli. Towards topological- geometrical theory of group equivariant non non-expansive operators for data analysis and machine learning. Nature Machine Intelligence, (9):423-433, 2019.

---

Under review as a conference paper at ICLR 2021
Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. Natural language processing (almost) from scratch. Journal of machine learning research, 12:2493-2537, 2011.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 248-255. leee, 2009.
Yonatan Dukler, Wuchen Li, Alex Lin, and Guido Montufar. Wasserstein of wasserstein loss for learning generative models. In Proceedings of Machine Learning Research, volume 97, pp. 1716- 1725. PMLR, 2019.
Daniel Gabay. Minimizing differentiable function over differential manifold. Journal of Opti- mization Theory and Applications, 37(2):177-219, 1982.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural infor- 2014.
mation processing systems, pp.
lan J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In International Conference on Learning Representations, 2015.
Geoffrey Hinton, Li Deng, Dong Yu, George E Dahl, Abdel-rahman Mohamed, Navdeep Jaitly, Andrew Senior, Vincent Vanhoucke, Patrick Nguyen, Tara N Sainath, et al. Deep neural networks for acoustic modeling speech recognition: The shared views of four research groups. IEEE Signal processing magazine, 29(6): 2012.
Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. In International Conference on Learning Representations, 2014.
Durk P. Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1xl convolutions, In Advances in neural information processing systems, pp. 10215-10224, 2018
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska- Barsskikka, al. Overcom- ing catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):352133526 2017.
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Ci (canadian institute for advanced re- search). URL http://www..cstoromto cdf//rir//iffrr.umm http://c.o.oooooocccr///aarrrrre
2010.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. Imagenet classification with deep convo- lutional neural networks. In Advances in neural information processing systems, pp. 1097-1105, 2012.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278-3324 1998.
James Martens. New insights and perspectives on the natural gradient method. Journal of Machine Learning Research, 21 21(146):1-76. 2020.
Frank Niclsen. An elementary introduction to information geometry. 1808::880..822 2018.
arXiv preprint
Chris Olah, Alexander Mordvintsev, and Ludwig Schubert. Feature visualization. Distill, 2017. doi: /ffo:a/illppii00///aaaeeeeeeiii
10.2391 1.331///iiiill.0000
William Peebles, John Peebles, Jun-Yan Zhu, Alexei A. Efros, and Antonio Torralba. The hessian penalty: A weak prior for unsupervised disentanglement. In Proceedings of European Conference on Computer Vision (ECCV), 2020.
Stefan Sommer and Alex M Bronstein. Horizontal flows and manifold stochastics in geometric deep learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020.
10

---

Under review as a conference paper at ICLR 2021
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethink- ing the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2818-2826 2016.
Loring W. Tu, An Introduction Manifolds. Universitext, Springer, 2008
Aaron Van den Oord, Nal Kalchbrenner, Lasse Espeholt, Oriol Vinyals, Alex Graves, et Con- ditional image generation with pixelenn decoders. In Advances in neural information processing systems, pp. 4790-47999 2016.

---

Under review as a conference paper at ICLR 2021
A APPENDIX: FROBENIUS THEOREM
For the reader's 's convenience, we collect here few facts regarding differentiable manifolds, for more details see Abraham al. (2012). Tu (2008).
Let be a differentiable manifold. Its tangent space T, . at point I E M can be effectively defined the vector space of derivations at I, T.M {E, a'a,}. after we choose chart around z. A vector field X on is a function x + X₂ E M assigning to any point of x E M tangent vector Xy, An integral curve of the vector field X is map (-e,t) M, such that c(0) and c(t) Xettj. for all & (-c,c), R.
For example, in R22
define the vector field:
X (z, v) yy z.,,
One may also view X assigning to point p (z,y) in R2 the vector (-v,z) applied at p. The integral curve X at (1.0) is the circle c(t) = (cos(t), sin()),, (-sin(t), cos(t)) Xe(t)- More in general, the integral curve of X generic point p is the circle centered at the origin and passing through p. If TM H. M T (disjoint union) denotes the
(-T, TT) in fact (t(t)
Figure 7: Vector field X = y², + xd, in IR? and its integral curves.
langent bundle, we also call the vector field X a section of TM.
Since T.M . is the vector space of derivations
differentiable functions defined neighbourhood
of x, we can write (x(f); this is the result of the application of the derivation X₂ E T.M to the function f E C (U). where (U)) denotes the differentiable functions f U E. x E U open in M. As varies in U, Xf)()) defines a function, which we denote with X()).
We call (M) the smooth vector fields on M, that is, those vector fields expressed in local coordi- nates as X= = E; (z),,, with U - R smooth functions, x E U, U open in M.
We can define a bracker on the space of vector fields as follows:
[X,Y]() (()()) Y(X()),
C X
[ is bilinear and satisfies:
X, YJ -Y, X] (antisymmetry);
2. XX |Y, Z]] [Y, [Z X]] [X. 0 (Jacoby identity).
We are ready to define distributions, which play a key role into our treatment.
Let M be differentiable manifold. We define distribution D of rank & assignment:
- D2 C T,M,
Vx E M
where Da is a subspace of T₂ M of dimension We say that smooth if at each point x E M, there exists an open
x E U. Dy is spanned by smooth vector fields on for all y E U.
We say that a distribution is integrable if for any E M there exists local submanifold N C U, U open in M, x N. called a local integrable manifold such that T&N D. for all E N. When such N exists globally, we say we have a foliation of M, that is we can write M as the disjoint union of integrable submanifolds, all of the same dimension and immersed in M. Each integrable submanifold of the foliation is called leaf. For example in IR3 {(0,0, 0, 0)}. the distribution:
p (x,, 2) Dr (span(x0, + vd, + =0.))=
(11)
12

---

Under review as a conference paper at ICLR 2021
is integrable: each point p. Dp is the tangent plane to sphere centered at the origin and passing through p. Hence we can write the space [(0, 0, 0)} as the disjoint union of spheres: each sphere a leaf of the foliation thus obtained. We may also say that RR& {(0, O, 0)} foliates as the disjoint union of spheres. Another example is given in Figure 7 showing how the distribution generated by the vector field X foliates I \ {0, 0)} as the disjoint union of circles.
We say that distribution is involutive if for all vector fields X,Y E D we have [X, Y] E D.
Frobenius Theorem establishes an equivalence between these two key properties of a distribution, namely integrability and involutivity, thus giving us effective method to establish when a distri- bution gives a foliation of M into disjoint submanifolds.
Theorem A.1. Frobenius Theorem Let M be a differentiable manifold and D a distribution on M. Then D is involutive if and only if it is integrable. If this occurs, then there exists foliation on M, whose leaves are given by the integrable submanifolds of
Proof. See Abraham al. (2012) Section 4.4.
Frobenius Theorem tells us that an involutive distribution on M defines at each point p submanifold N of dimension & of M. This means that, locally, we can choose coordinates (FT))..,;xx)) for M, so that, in the chart neighbourhood of p. the submanifold N is determined by equations 2k++1 = ck-1..... 2n, Cz. where c, are constants. We have then that (I1,... ......... x&& are local coordinates for the submanifold N around p. Notice that the vector fields associated to these particular coordinates X; 0r, verify stronger condition than involutivity, namely [X,, X,] = O. Furthermore, the proof of Frobenius theorem is constructive: it will give us the vector fields X, and the coordinates z; as their integral curves. Let us see in a significant example, what this construction amounts to.
We express the distribution D defined in [1 explicitly as:
= (I, U. 2) D, = (span(xd,, + +yy + =0=3)+
(12)
spanfX = yd, - zd,, Y = =z0; =22z)
As one can check [X,Y], we transform
-y0 + z0, E Dr for all E Rd {(0,0,0)). With Gauss reduction
1/ x x 0 - - 6
0 -1/
0 1/2
0
One can then verify that X2] = 0 for X1 02 (z/2))2, X2= (x,)
0y
(y/z)0,.
The calculations of this example are indeed a prototype for the treatment in the general setting: given a local basis for an involutive D, that is a of vector fields XI,...X,, which are linearly independent in a chart neighbourhood, using Gauss algorithm, we can transform them into another basis Y1 ... Ys with the property [Y, Yý] 0. The integral curves of the Y,'s will then give us the coordinates for the leaf manifold in the given chart.
B APPENDIX: PROOFS
In this appendix we collect the proof of the mathematical results stated in our paper. Recall the two key definitions of local Fisher and data matrices:
F(z, w) Eyyx (log(p(ylz, w)) V
w))
(13)
G(æ, w) = Eyypl V (log(p(vlz, w)) V. (log(p(y/r, w)
(14)
We start with the statement and proof of Proposition 2.1 Proposition B.1. Let the notation be in Section 2 Then:
1. ker F(x, w) = span { V logg w) < sis CJ+
kcr G( w) span VZ logpi w Zis
2. rank F(x, w) < C. rank G(r,w) C.
13

---

Under review as a conference paper at ICLR 2021
Proof. We prove the results for F(z, w), the proofs for G(x, w) are akin.
We show that ker F(x, w) E span (og w) I 2 C)':
E ker Fxx w) + u u 0 uyup (V ~ log logp(y))), u} 0
(15)
( logg: = vi = ,,...,C. C.
On the other hand, if u span V w log
< с) then ker F(x, w): F(z, wz = Ey-p w (v) log (3/22), u)] = O. From 2.1 we know that rank F F(x, w) I C. Equation (4) allows us to conclude the proof.
(16)
We now go to the main mathematical result of our paper, which is the key to provide with manifold structure the space of data.
Theorem B.2. Let w be the weights of deep ReLU neural network classifier, p given by softmax, G(I, w) the local data matrix. Assume G(x, w) has constant rank. Then, there exists a local sub- manifold N C. R", N, such that its tangent space at 2, T&N ker(G(zz, -))= for all N.
Proof. By Frobenius Theorem, we need to check the involutivity property (10) for the distribution
x + D2 (er(G((r, w) on R". Since by Prop. 2.1
D3 = ker (xr, w))+ = span V log 11 kskss
we only need to show that:
< pc) w)), v log (vlo, w)] spant 2 logg:(y/z,, 1 < k cs By standard computations, we see that:
V (1 x w), Va logp;(vlz, w)] (log (yl w) V log (3/2, w)+ H[(llgps (vl=,, w) v po w), where H(ʃ) denotes the Hessian of a function f. Here we are using the fact that
(17)
Σ ad Σ b,0,1 = Σ
(a,0,by b,0,a,),,
2
With some calculations, we have:
[: (H(()(y) w)) V, logpi(ylz, w) (V, (ogp( w)
(18)
and
Vapi(yz,,)) E Poll, w). pa(y W)) k=1
(19)
where s is the score function and we make use of the fact pylz, w) is given by sofimax. Notice Pr D,yPi,yl w) = (, - Pa (2)2,, w)), where du the Kronecker delta. By (88 and 19):
(yl w)) = (x = Va [p:(y (6 Pk (y/z, w)M sT,
(20)
where we use the fact that 8 is piecewise linear so that its Hessian is zero, where it is defined. Hence:
(yp(,, w)) Veplyyla () ( (v
Ps
Σ (yll
(21)
Pr(y) w)
k=1
Now, in view of (17). using 18) and (1 we compute the expression:
Hi(log Hi(logpi(ylz w)) = (yl w)
Vapi(vla, ps Pr(yla, w) p
Σ V (ul
Va log w) (Va logpi(u(e, u))²
Σ
k=
(22)
14

---

Under review as a conference paper at ICLR 2021
Hence:
C Hoog (.(y, V (ogg:::: = Σ ()) V
Py py w).
Since (V,s3) V. a Py w) is a scalar, we obtain Hl w))V log (y/a, w) is lin- ear combination vectors V.Px (vlar, ar w), and therefore a linear combination of V log pk (yla, w). The same holds for [V log Pa (V z, w), V log (ylaz, w)] wanted to show.
C APPENDIX: CIFAR- 10 EXPERIMENTS
We perform some experiments with the CIFAR- 10 dataset and neural network similar to VGG-11.
Figure 8: Horizontal paths between images in CIFAR-10 test set.
15


Reviews:
[{"review_id": "g6-G-Qjelre", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this work, the authors showed that deep ReLU networks can model the low dimensional manifold structure of the dataset. The authors first define a local data matrix G which is analogous to Fisher matrix. Then they proved that the tangent space of the data manifold is spanned by the eigen vectors of G corresponding to non-zero eigen values. The authors visualize this data manifold on MNIST data. \n\nBelow, I present some of the key points (strengths/weaknesses) for this paper. The concrete theoretical result to characterize the data manifold based on the span of the eigen vectors of the data matrix is novel and an important result. On the other hand, simple demonstration on MNIST may not be sufficient to conclude the effectiveness on other natural images, e.g., cifar, imagenet. The authors in [1] argued that for simple dataset like MNIST, the lower dimensional representation may be flat but for natural images, the lower dimensional representation may possibly be curved. This makes one wonder the applicability of such kind of path analysis for natural images. Although, I believe the theoretical contribution is nice, the lack of empirical validation/evaluation weakens the scope of the paper, which justifies the rating ````\"6\". \n\n[1] https://papers.nips.cc/paper/8843-intrinsic-dimension-of-data-representations-in-deep-neural-networks.pdf\n\n1. It is always a better idea to explain rationale and motivation of a Proposition before stating and proving it, e.g., while reading Proposition 1 for the first time, I have no idea the need for looking at kernel space!\n\n2. The main theorem stating that for deep ReLU network, then the submanifold we get with the tangent space as orthogonal space of the kernel of the data fisher matrix is I believe a very important result. \n\n3. Although I am very impressed with the theoretical result presented in the paper, in practice constancy of rank may not be a valid assumption and as mentioned in section 4 that towards the end the fisher matrix becomes close to a null matrix, the impracticality of the paper comes from the usage of partially trained model. For example, it is not realistic to assume (1) a given network is partially trained (2) not too deep so doesn't converge fast. These in my opinion are bug assumptions and is a bottleneck for the applicability of this paper.\n\n4. Although the justification of not using retraction is meaningful, not sure the need for normalizing gradient, is it to get a ``good grip'' on learning rate?\n\n5. The implication of Fig. 4 showing that the data manifold may include points outside our training/test data is a good observation but I believe things are too simplistic and \"too good\" for MNIST data. Any comment on dataset with natural images like Cifar10, ImageNet etc.?\n\n6. How the authors compute G(x,w), i.e., Jacobian?\n\n7. In algorithm 1, line 4, where is j used, is it to compute D_x?", "extra_scores": {}, "extra_text": {}}, {"review_id": "bcCkCT6lxe", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper provides an information geometric view on deep ReLU neural network\nclassifiers which use the softmax function. For a fixed model it defines the\n``local data matrix'' based on gradients \\( \\nabla _x \\log p(y|x, w) \\) of\nconditional log probabilities combining ideas from both scores and the Fisher\ninformation matrix. This matrix gives rise to a Riemannian manifold with a\nspecific foliation of interest. Experiments using the MNIST dataset suggest\nthat valid data live in a single leaf.\n\nThe finding that involutive property is satisfied for the distribution D_x\nconnected to the localhost Fisher matrix G is to me quite surprising and an\ninteresting result. With the constant rank assumption, this indeed implies a\nfoliation of the data space. It would be nice if the authors could make it\nclearer what is the interpretation of the local Fisher matrix, i.e. what it\nactually models. This is attempted experimentally but to a lesser degree\ntheoretically.\n\nExperimental results are presented to validate different hypotheses:\nExperiment 1 attempts to validate the assumption of the local data matrix\nhaving constant rank. The authors observe that it breaks for models reaching\nconvergence. The quantitative relation to Figure 1 is not clear to me and I\ndid not find the experiment readily reproducible from the source code\nprovided. The remaining experiments look at paths between pairs of images\nfrom the MNIST test set and images moved along a path. It is not clear how\nthe 1-2 examples presented for each of the experiments have been selected.\nConsidering them as ``remarkable confirmation of [the] theoretical findings''\nof an impossibility statement feels like an overstatement.\n\nIt would be nice if the authors could comment on how an example where the\ndata is distributed to fill the entire data space can be foliated into C-1\nleafs with the data being contained in one of those leafs. This leaf would\nthen need to fill the entire data space. Is this a case where the constant\nrank assumption wails?\n\nThe written exposition does not feel ready for presentation yet. Smaller\nmistakes (a/an, plural/singular, lie/lay) and notational issues (e.g. missing\nnabla in the line before eq. (4)) impede readability. The introduction\nignores generative models like (deep) Boltzmann machines or the Helmholtz\nmachine and is partially inaccurate (generative models providing an explicit\ntransformation). The paper's goal or contribution does not become clear from\nthe abstract.\n\nOverall, in my view, the paper does contain promising contributions, however,\nin its current state, it is somewhat premature and the presentation is\n(slightly) below the acceptance threshold for ICLR.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "YfXHcAe47bA", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: The authors propose a so called data matrix that is induced in the input space of a deep neural network classifier. This matrix is similar to the Fisher-Rao metric, but for the input and not the parameters of the model. The analysis of this matrix shows that the classifier induces in the input space a specific structure, which the authors study. Constructive experiments are used in order to empirically verify the claims.\n\nComments:\n1) In general, I find the paper a bit hard to follow and understand. In my opinion, the overall the writing of the paper can be improved. Some comments and suggestions that I believe will help:\n- The proofs can be moved in the appendix to save space.\n- The clarity of the paper can be improved. In general, be more explicit and provide clearer explanations.\n- The motivation of the paper is not very clear to me. What is the \"problem\" that you aim to solve? Or what is the \"purpose\" of doing the proposed analysis. These goals should be more clear.\n- The coherence of the paper can be improved.\n- There are a lot of terms which are not defined explicitly. For instance, what is the foliation, the data leaf and its dimension?\n- I suggest the authors to include some figures, which will help the reader to understand the proposed idea.\n- I believe that the introduction is a bit unclear. \n\n2) In general, after reading the paper, I am not sure if I can understand what is the main motivation? What is exactly the data leaves and the foliation?\n\n3) I guess that the degeneracy of the matrix comes from the fact that, when the classifier is confident around a point only 1 class is selected and the probabilities do not change. So what is the purpose of under-training a model e.g. using 1 SGD step? I understand that this is done in order to avoid degenerate matrices, but why is interesting to study such a non-trained model?\n\n4) How the submanifolds in the input space can be seen intuitively? I think an image will help a lot the understanding. Similarly, the distribution of Eq. 11 is not very clear.\n\n5) I think that the proposed idea may be interesting. However, I believe that the current version of the paper needs to be improved such that to make the idea accessible. In my opinion, the current stage of the paper is not ready to be published.\n\n===== After rebuttal =====\n\nI appreciate the fact that the authors took into account our comments and improved their manuscript (+1), however, I think that there is still space for improvement.\n\nMy main concern, is the fact that we need the model to not be \"fully trained\" to do some analysis about \"what the model learns\". Therefore, regarding the data leaf, I am not sure if the trajectory that connects two points is actually moving on a \"data manifold/leaf\" or simply in the data space. Anyways, the behaviour of the metric away from the given data is kind of arbitrary since extrapolation analysis in neural networks is quite difficult. So the most crucial assumption is that the (learned) metric gives meaningful directions to move in the data space. In the experiments this seems to be the case \"roughly\", but the result could have been the same simply by the linear interpolation (which I think is missing). Similarly, if we just move linearly in a random direction, probably noise will appear gradually.", "extra_scores": {}, "extra_text": {}}, {"review_id": "5cmtO9ziMfV", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Update after response:\nWhile I appreciate the authors' attempt to address my concerns, the fact that model is required to not be fully trained is concerning. It was in this context that I suggested label smoothing - that training on smoothed labels might address a sparse G matrix, but it seems like this point was not communicated clear enough by me and not understood by the authors. \n\nThe authors suggest some applications of the model centric view of the data, but do not present any experiments regarding these applications. I believe the paper might be more convincing if those experiments are added in the next version of the paper.\n\nAt this time however I will still have to vote for rejection.\n\nOriginal Review:\n\nThis paper describes a manifold construction of the space of inputs to a deep ReLU network, by defining a metric using the gradient of the network's loss function with respect to its input. The paper uses the Frobenius Theorem to show that every point in the input space R^n can be associated with a submanifold which constitutes \"the model's view of the data.\"\n\nMy main concern with this work is that the import of this work is not clear. At the current stage, the authors have only identified a foliation of R^n (not the data manifold, since that is presumably only a low dimensional manifold of R^n). Do the authors envision using their manifold construction as a generative model? Or will this view of the input space assist in identifying adversarial examples or coming up with defenses? Can this approach help in identifying structures within the data?\n\nWhile the use of Frobenius Theorem is new, this line of research of visualizing the \"manifold\" of data \"preferred\" by the model goes back to the deep dream visualizations from google and other efforts to understand the representations learned by deep networks. The authors do not refer to that line of work, and I would be curious to learn how they see their work differing from visualizations of features. Perturbing images along gradients of the log likelihood is also a common technique (Fast Gradient Sign Method and Projected Gradient Descent) to generate adversarial attacks for models. Can the authors use their analysis to identify submanifolds of adversarial examples?\n\nThe authors also show results only on the MNIST dataset, while these are promising, atleast extending their results to CIFAR10,100 will lend more support to their ideas. I also do not understand the difficulty in using fully trained models to perform their experiments. If there is an issue with a sparse G matrix then I suggest the authors try techniques like label smoothing to prevent this. If this cannot be done then that might suggest deeper issues with the paper, since the claim of describing the \"model-centric data manifold\" cannot be made if the fully trained model is not used.\n\nWhile I have not studied the steps of Theorem 3.1 completely, it would help to add a step to equation 13 explaining how the Hessian came into the picture. \n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper defines a "local data matrix" (inspired from local Fisher matrix) and uses it to obtain a foliation in the data space. This provides a lens to view the data space from model's perspective. While the idea is interesting, reviewers have two main concerns from the reviewers which are not fully addressed in the author response: 
(i) The method works with partially trained model (1 epoch for MNIST) and it's not clear how the observations made in the paper extend to fully trained models, 
(ii) The motivation and application of the proposed model-centric view of data space needs more work - it will be good to think of some applications where this view can help. 

I encourage the authors to consider the suggestions from the reviewers (e.g, R3 suggested label smoothing for (i)), and submit a revised version to a future venue. 
Decision: Reject