Manifold Learning via Foliations, and Knowledge Transfer

## Abstract

Abstract Understanding how real data is distributed in high dimensional spaces is the key to many tasks in machine learning. We want to provide a natural geometric structure on the space of data employing a deep ReLU neural network trained as a classifier. Through the data information matrix (DIM), a variation of the Fisher information matrix, the model will
discern a singular foliation structure on the space of data. We show that the singular points of
such foliation are contained in a measure zero set, and that a local regular foliation exists almost everywhere.
Experiments show that the data is correlated with leaves
of such foliation.
Moreover we show the potential of our approach for knowledge transfer by analyzing the spectrum of the DIM to measure distances between datasets.

## 1 Introduction

The concept of manifold learning lies at the very heart of
the dimensionality reduction question, and
it is based on the assumption that we have a natural
Riemannian manifold structure on the space of data
[, , , ]. Indeed, with such
assumption, many geometrical tools become readily available for machine
learning questions, especially in connection with
the problem of knowledge transfer [, ], as
geodesics, connections, Ricci curvature and similar
[].
In particular, the fast developing field of Information
Geometry [, ], is now providing with the techniques
to correctly address such questions [].

However, the practical situation, arising for example in classifying
benchmark datasets as MNIST [], Fashion-MNIST [] and similar,
with CNNs (Convolutional Neural Networks), is generally more complicated
and does not allow for such simple description, at least in the cases most relevant for concrete applications, and calls for more
sophisticated mathematical modeling.

Machine learning models can be roughly divided into two categories:
classifier and generative models. In particular, Deep Learning
vision classifier models, via their intrinsic hierarchical
structure, offer a natural representation and implicit organization of the input data []. For example, the authors of [] show how a multilayer
encoder network can be successfully employed to transform high
dimensional data into low dimensional code, then to retrieve it via a “decoder”.

Such organisation strikingly mirrors the one occurring in the
human visual perception, as also observed in the seminal work
[], and later an inspiration for the
spectacular successful employment of CNNs for supervised classification tasks
[].

In this paper, we want to provide the data space
of a given dataset with a natural geometrical structure,
and then employ such structure to extract key information.
We will employ a suitably trained
CNN model to discern a foliation structure
in the data space and show experimentally
how the dataset our model has been trained with
is strongly correlated with its leaves, which are submanifolds of the data space.
The mathematical idea of foliation is quite old (see [, ]
and refs therein); however, its applications in control theory (see [] and refs therein)
via sub-Riemannian geometry and machine
learning have only recently become increasingly
important [, ].

The foliation structure on the data space, discerned by our model,
however, is non-standard and presents singular
points. These are points admitting a neighbourhood
where the rank of the distribution, tangent at each point to a leaf of the foliation,
changes. Moreover, in the presence of typical
non linearities of the network, as ReLU, there are also non smooth points, thus hindering in
such points the smooth manifold structure itself of the leaf.
However, we prove that both singular and smooth points are a measure zero
set in the data space, so that the foliation is almost everywhere regular
and its distribution well defined.
As it turns out in our experiments, the samples belonging to the
dataset we train our model with are averagely close
to the set of singular points. It forces us to
model the data space with a singular foliation that
we call learning foliation in analogy with the learning manifold.
Applications of singular foliations were introduced in connection with control theory [];
their study is currently an active
area of investigation [].
As we shall
show in our experiments, together with their distributions, singular foliations provide with an effective tool to single
out the samples belonging to the datasets the model was trained with and at the same time discern
a notion of distance between different datasets belonging to the same data space.

Our paper is organized as follows. In Sec. 2, we recap the
previous literature, closely related to our work.
In Sec. 3, we start by recalling some known facts about Information
Geometry in 3.1. Then, in Sec. 3.2 we introduce the data information
matrix (DIM), defined via a given model,
and two distributions $\mathcal{D}$ and $\mathcal{D}^{\perp}$ naturally associated
with it, together with some illustrative examples elucidating the
associated foliations and their leaves. In Sec. 3.3 we introduce singular distributions
and foliations and then we prove our main theoretical results
expressed in Lemma 3.4 and Theorem 3.6.
Lemma 3.4 studies the singularities of the distribution $\mathcal{D}$.
Theorem 3.6 establishes that the singular points
for $\mathcal{D}$ are a measure zero set in the data space. Hence our foliation, though singular, acquires geometric significance.
Finally in Sec. 4 we elucidate our main results with experiments.
Moreover, we show how the foliation structure and its singularities can be exploited
to determine which dataset the model was trained on, and its distance
from similar datasets in the same data space.
We also make some "proof of concept" considerations
regarding knowledge transfer to show the potential of the mathematical singular
local foliation and distribution structures in data space.

## 2 Related Works

The use of Deep Neural Networks as tools for both manifold learning
and knowledge transfer has been extensively investigated.
The question of finding a low dimensional
manifold structure (latent manifold) into high dimensional
dataset spaces starts with PCA and similar methods to reach the
more sophisticated techniques as in [, ],
(see also [] for a description of the most popular techniques
in dimensionality reduction and
[] for a complete bibliography or the origins of the subject).
Such understanding was applied towards the knowledge transfer questions in
[, ] and more recently []. Though this is not our approach,
it is important to mention that one
important technique of knowledge transfer is via shared parameters
as in [].
The idea of using techniques of Information Geometry for
machine learning started with []. More recently, it was used
for manifold learning
in [] (see also refs therein). Employing
foliations to reduce dimensionality is not per se
novel.
In [], the authors introduce some foliation structure in the data space of a neural network, with the aim of approaching dimensionality reduction.
In [], orthogonal foliations in the data
space are used to create adversarial attacks, and provide
the data space with curvature and other Riemannian notions to analyse such attacks.
In [] invariant foliations are employed to produce a
reduced order model for autoencoders. Also, in considering singular
points for our foliation, we are led to study the singularities of a neural
network. This was investigated, for the different purpose of network expressibility
in [].

## 3 Methodology

### 3.1 Information Geometry

The statistical simplex consists of all probability
distributions in the form $p(y|x,w)=(p_{1}(y|x,w),\dots,p_{c}(y|x,w))$,
where $x$ is a data point, belonging to a certain dataset $D\subset\mathbf{R}^{d}$,
divided into $c$ classes, while $w\in\mathbf{R}^{N}$ are
the learning parameters [, , ].
We define the information loss as:
$I(x,w)=-\log(p(y|x,w))$ and the Fisher information matrix (FIM) [] as

$$ $F(x,w)=\mathbf{E}_{y\sim p}[\nabla_{w}\log p(y|x,w)\cdot(\nabla_{w}\log p(y|x,w))^{T}]$ (1) $$

In analogy
to (1) we define the Data information matrix (DIM):

$$ $D(x,w)=\mathbf{E}_{y\sim p}[\nabla_{x}\log p(y|x,w)\cdot(\nabla_{x}\log p(y|x,w))^{T}]$ (2) $$

As noted in [], some directions in the parameter or data space, may be more
relevant than others in manifold learning theory.
We have the following result [], obtained with a direct calculation.

###### Proposition 3.1 .

The Fisher information matrix $F(x,w)$ and
the data information matrix
$D(x,w)$ are positive semidefinite symmetric matrices. Moreover:

$$ $\begin{array}[]{c}\ker F(x,w)=(\mathrm{span}_{i=1,\ldots,c}\{\nabla_{w}\log p_{i}(y|x,w)\})^{\perp},\\ \\ \ker D(x,w)=(\mathrm{span}_{i=1,\ldots,c}\{\nabla_{x}\log p_{i}(y|x,w)\})^{\perp}.\end{array}$ (3) $$

where the orthogonal is taken with respect to the euclidean product.
In particular $\mathrm{rank}\ F(x,w)<c$ and $\mathrm{rank}\ D(x,w)<c$.

Notice that $F(x,w)$ is a $N\times N$ matrix,
while $D(x,w)$ is a $d\times d$ matrix, where
in typical applications (e.g. classification tasks) $N,d>>c$.
Hence, both $F(x,w)$ and $D(x,w)$ are, in general, singular matrices,
with ranks typically low with respect to their sizes, hence
neither $F(x,w)$ nor $D(x,w)$ can provide meaningful metrics
respectively on parameter and data spaces. We shall now focus on
the data space.

The result (3)
suggests to consider the distribution $\mathcal{D}$:

$$ $\begin{array}[]{c}\mathbf{R}^{d}\ni x\mapsto\mathcal{D}_{x}:=\mathrm{span}_{i=1,\ldots,c}\{\nabla_{x}\log p_{i}(y|x,w)\}\subset T_{x}\mathbf{R}^{d}\end{array}$ (4) $$

where we assume the learning parameters $w\in\mathbf{R}^{N}$ to be fixed as $x$ varies.
In general a distribution on a manifold $M$ assigns to each point a
subspace of the tangent space to $M$ at that point (see [, ] for more
details), in our case $M=\mathbf{R}^{d}$.

###### Proposition 3.1 .

### 3.2 Distributions and Foliations

Whenever we have a distribution on a manifold $M$,
it is natural to ask whether it is integrable.
A distribution $\mathcal{C}$ on $M$ is integrable if for every
$x\in M$, there exists a connected immersed submanifold $N$ of $M$, with $T_{y}N=\mathcal{C}_{y}$ for all $y\in N$.
In other words, the distribution defines at each point the tangent space
to a submanifold. Whenever a distribution is integrable, we have a foliation,
that is, $M$ becomes a disjoint union of embedded submanifolds
called the leaves of the foliation. If $\dim\mathcal{C}_{x}=\dim\mathcal{C}_{y}$ for
all $x,y\in M$ we say that the foliation, or the corresponding
distribution, has constant rank. Fig. 1 illustrates a constant rank foliation and its
orthogonal complement foliation, with respect to
the euclidean metric in $M=\mathbf{R}^{3}$ (the ambient space).

Figure: Figure 1: Foliations: $\mathcal{L}_{\mathcal{D}}$ and $\mathcal{L}_{\mathcal{D}^{\perp}}$ denote the set of the leaves in distributions $\mathcal{D}$ and $\mathcal{D}^{\perp}$ respectively.
Refer to caption: x1.png

Under suitable regularity hypothesis, Frobenius theorem [, ]
provides with a characterization of integrable distributions.

###### Theorem 3.2 .

(Frobenius Theorem).
A smooth constant rank
distribution $\mathcal{C}$ on a real manifold $M$
is integrable if and only if it is involutive,
that is for all vector fields $X,Y\in\mathcal{C}$, $[X,Y]\in\mathcal{C}$.
(^1^11If $X$, $Y$ are vector fields on a manifold, we define their Lie bracket as $[X,Y]:=XY-YX$.)

To elucidate this result in Fig. 4
we look at two examples of foliations of 1-dimensional distributions, obtained
from the distribution $\mathcal{D}$ in (4) of a neural network trained on the Xor function, with non linearities GeLU and ReLU(^2^22We look only at smooth points, which form an open set.). Notice that since in both cases $\mathcal{D}$ is 1-dimensional, then it is automatically integrable (bracket of vector fields being zero).
We have the following result; the proof is a simple calculation based
on the Frobenius Theorem [].

###### Proposition 3.3 .

Let the notation be as above.
For an empirical probability $p$ defined via a deep ReLU neural network the
distribution $\mathcal{D}$ in the data space:

$$ $x\mapsto\mathcal{D}_{x}=\mathrm{span}_{i=1,\ldots,c}\{\nabla_{x}\log p_{i}(y|x,w)\}$ $$

is locally integrable at smooth points.

We call the foliation resulting from Prop. 3.3
the learning foliation. Its significance is clarified
by the following examples.

Figure: Figure 2: Moving according to $\mathcal{D}$.
Refer to caption: Images/06311_tangent_direction_-start_no_clamp_color.png

Figure: Figure 3: Moving according to $\mathcal{D}^{\perp}$.
Refer to caption: Images/06311_noise_-start_no_clamp_color.png

As an illustration of the distributions $\mathcal{D}$ and $\mathcal{D}^{\perp}$
(orthogonal
computed with respect to the euclidean metric in $M=\mathbf{R}^{d}$), in
MNIST, we present Fig. 2 and Fig. 3. We notice that, while moving
in $\mathcal{D}$ from one data point
, the model predicts a meaningful label,
while moving in $\mathcal{D}^{\perp}$, the model maintains high confidence on the label,
though not in line with the image.
Both Fig. 2 and Fig. 3 were
obtained by projecting the same direction on $\mathcal{D}$ and $\mathcal{D}^{\perp}$ at each step.
Notice that the phenomena we observe
above are due to the fact that the neural network output probabilities are invariant by moving in the kernel of the DIM, that is the distribution $\mathcal{D}^{\perp}$.

For a model with GeLU non linearity however, one can see
experimentally that we do not have the involutivity property anymore
for the distribution $\mathcal{D}$ (see Table 2).
Hence,
there is no foliation whose leaves fill the data
space, naturally associated to it via Frobenius Theorem. To see this
more in detail,
we define the space $\mathcal{V}^{D}_{x}$ generated by $\mathcal{D}_{x}$ and the Lie brackets of their generators:

$$ $\begin{array}[]{c}\mathcal{V}^{D}_{x}=\mathrm{span}\{\nabla_{x}\log p_{i}(y|x,w),[\nabla_{x}\log p_{j}(y|x,w),\nabla_{x}\log p_{k}(y|x,w)],i,j,k=1,\dots,c\}\\ \\ \end{array}$ (5) $$

**Table 1: Involutivity of the distribution $\mathcal{D}$**
| Non linearity | dim $\mathcal{D}_{x}$ | dim $\mathcal{V}^{D}_{x}$ |
| --- | --- | --- |
| ReLU | 9 | 9 |
| GeLU | 9 | 44.84 |
| Sigmoid | 9 | 45 |

In Table 2 we report averages of the dimensions of the
spaces $\mathcal{D}_{x}$, $\mathcal{V}^{D}_{x}$ for a sample of 100 points $x\in\mathbf{R}^{d}$.
The non involutivity of the distribution
is deduced from the fact the dimension increases when we take the
space spanned by the distribution and the brackets of its generators.
As we can see, while for the ReLU non linearity
$\mathcal{D}$ is involutive and we can define the learning
foliation, the brackets of
vector fields generating $\mathcal{D}$ do not lie in $\mathcal{D}$ for the GeLU and sigmoid non linearities.
Consequently, there is no foliation and the sub Riemannian
formalism appears more suitable to describe the geometry in this case.
We shall not address the question here.

###### Theorem 3.2 .

###### Proposition 3.3 .

### 3.3 Singular Foliations

A foliation on a manifold $M$ is a
partition of $M$ into connected immersed submanifolds, called leaves. A foliation is
regular if the leaves have the same dimension, singular otherwise.
Notice that the map $x\mapsto\mathrm{dim}(\mathcal{L}_{x})$, which associates to
$x\in M$ the dimension of its leaf $\mathcal{L}_{x}$, is lower semi-continuous, that is,
the dimensions of the leaves in a
neighbourhood of $x$ are greater than or equal to $\mathrm{dim}(\mathcal{L}_{x})$ []. Whenever we have an equality, we say that $x$ is regular, otherwise
we call $x$ singular. It is important to remark that, adhering to the
literature [], the terminology singular point
here refers to a point that has a neighbourhood where the leaves have
non constant dimension. We can associate a distribution to a foliation by associating to each point, the
tangent space to the leaf at that point.
Such distribution has constant rank if and only if the foliation is regular. Frobenius Theorem 3.2 applies only to the case of constant
rank distributions,
however, there are results extending part of Frobenius theorem to non constant
rank distributions. In [] and [], the authors give sufficient
conditions for integrability in this more general setting
(see [] for correct attributions on statements and proofs).

In the practical applications we are interested in, foliations may also have non smooth points,
that is points where the leaf through the point, belonging to the foliation, is not smooth. For example, Fig. 4
shows a Xor network with ReLU non linearity displaying both
singular and non smooth points.

Figure: (a) GeLU.
Refer to caption: x4.png

The singular point of each picture is in the center,
while non smooth points occur only for Xor with ReLU non linearity. As previously stated,
this is a degenerate case, where the involutivity and integrability
of $\mathcal{D}$ is granted automatically by low dimensionality,
though for (b) in Fig. 4,
we cannot define $\mathcal{D}$ at non smooth points.

###### Remark 3.1 .

Notice that Frobenius Theorem 3.2 and its non constant rank
counterparts, apply only in the smoothness hypothesis, while for applications, i.e. the case of ReLU
networks, it is necessary to examine also non smooth foliations.
We plan to explore the non smooth setting more generally in a forthcoming paper.

We now want to investigate further the singular points of the distribution $\mathcal{D}$.

Let $N(x)=p(y|x,w)=(p_{1}(y|x,w),\dots,p_{c}(y|x,w))$ denote the output of the neural network classifier and
$J_{N}(x)=(\nabla_{x}p_{i}(y|x,w))$ its Jacobian matrix. So $\nabla_{x}p_{i}(y|x,w)$ is the $i$-th row
(column) of $J_{N}(x)$.
One can see, by the very definition of $\mathcal{D}$ (4), that $\mathrm{rank}\mathcal{D}=\mathrm{rank}J_{N}$.
We assume $w$ to be constant, that is we fix our model. Let $P=\mathrm{diagonal}(p_{1}(y|x,w),\dots,p_{c}(y|x,w))$.

From now on we assume $N(x)=\mathrm{Softmax}\circ S(x)$
where $S(x)$ represents the score.
Then we have:

$$ $J_{N}(x)=(P-p^{t}p)J_{S}(x)$ (6) $$

where $p=(p_{1}(y|x,w),\dots,p_{c}(y|x,w))$ and $J_{S}(x)$ is the Jacobian of the score.

To study the drop of rank for $\mathcal{D}$ or equivalently for $J_{N}$, let us first look at the kernel
of $\left(P-p^{t}p\right)$.

###### Lemma 3.4 .

Let $E_{i}$ denote the vector with the $i$-th coordinate equal to one, and the others equal to zero.

$$ $\ker(P-p^{t}p)=\mathrm{span}\{(1,\dots,1)\}+\mathrm{span}\{E_{i},~\forall i\text{ such that }p_{i}=0\}$ (7) $$

###### Proof.

Let $u\in\mathbf{R}^{c}$. Then $u\in\ker(P-p^{t}p)$ if and only if
$p_{i}u_{i}-p_{i}\sum_{k}p_{k}u_{k}=0$. This is equivalent to:

$$ $p_{i}=0\quad\text{ or }\quad u_{i}-\sum_{k}p_{k}u_{k}=0\iff p_{i}=0\quad\text{ or }\quad\sum_{k}p_{k}(u_{i}-u_{k})=0$ $$

The inclusion $\supseteq$ is thus straightforward. To get the other inclusion, let $i_{0}$ denote the argmax of $u$. Then, $\sum_{k}p_{k}(u_{i_{0}}-u_{k})$ is a sum of non negative terms; hence to be equal to zero, must be $p_{k}(u_{i_{0}}-u_{k})=0$ for all $k$. Therefore, $u_{k}=u_{i_{0}}$ for all $k$ such that $p_{k}\neq 0$. This is enough to prove the direct inclusion $\subseteq$.
∎

We have the following important observation, that we shall explore more
in detail in our experiments.

###### Observation 3.2 .

Lemma 3.4 tells us that the rank of the distribution $\mathcal{D}$
or equivalently of $J_{N}(x)=(P-p^{t}p)J_{S}(x)$ is lower at points in the data space
where the probability distribution has higher number of $p_{i}\neq 0$.
Clearly the points in the dataset, on which our model is trained, are precisely the
points where the empirical probability $p$ is mostly resembling a mass probability distribution.
Hence at such points, we will observe empirically
an average drop of the values of the
eigenvalues of the DIM (whose columns generate $\mathcal{D}$),
compared to random points in the data space,
as our experiments confirm in Sec. 4. As we shall see, this
property characterizes the points in the dataset the model was trained with.

In our hypotheses, since the probability vector $p$ is given by a Softmax function, it cannot have null coordinates. Therefore, Lemma 3.4 states that $\dim\ker(P-p^{t}p)=1$ and that the kernel of $P-p^{t}p$ does not depend on the input $x$.
Thus, the drops in rank of $\mathcal{D}$ does not depend on $\ker(P-p^{t}p)$ and can only be caused by $J_{S}(x)$.

Now we assume that the score $S$ is a composition of linear layers and activation functions as follows:

$$ $S(x)=L_{W_{\ell}}\circ\sigma\circ\dots\circ\sigma\circ L_{W_{1}}$ (8) $$

where $\sigma$ is the ReLU non linearity, $L_{W_{i}}$ are linear layers (including
bias) and $\ell$ is the total number of linear layers.
We denote the output of the $k$-th layer:

$$ $f_{k}(x)=L_{W_{k}}\circ\sigma\circ\dots\circ\sigma\circ L_{W_{1}}(x)$ $$

Let us define, for a subset $U$ in $\mathbf{R}^{d}$:

$$ $Z_{U}=\left\{x\in U\text{ such that }\exists i,~x_{i}=0\right\}$ (9) $$

###### Lemma 3.5 .

Let $\mathcal{O}$ be the set of points in $M=\mathbf{R}^{d}$, admitting a neighbourhood
where the score function $S$ is non constant.
Then, the set of singular points of $J_{S}$, the Jacobian of $S$, is a subset of:

$$ $\bigcup_{k=1}^{\ell-1}f^{-1}_{k}(Z_{f_{k}(M)})\cap\mathcal{O}=\bigcup_{k=1}^{\ell-1}\bigcup_{i=1}^{\dim f_{k}(M)}\left\{x\mid f_{k}(x)_{i}=0\right\}\cap\mathcal{O}$ (10) $$

This set is the finite union of closed null spaces, thus of zero Lebesgue measure.

###### Proof.

A short calculation based on the expression of $S$ (8)
gives:

$$ $\begin{array}[]{lcl}J_{S}(x)&=&W_{\ell}J_{\sigma}\left(f_{\ell-1}(x)\right)W_{\ell-1}J_{\sigma}\left(f_{\ell-2}(x)\right)\dots J_{\sigma}(f_{1}(x))W_{1}\end{array}$ (11) $$

Notice that:

$$ $\left(J_{\text{RELU}}(x)\right)_{i,j}=\begin{cases}\delta_{i,j}&\text{if }x_{i}>0\\ 0&\text{if }x_{i}<0\end{cases}$ (12) $$

Hence, the set $Z_{U}$ represents the singular points of $J_{\text{ReLU}}$ on the domain $U$.

If $x_{0}\in\left\{x\mid f_{k}(x)_{i}=0\right\}\cap\mathcal{O}$, then, even if it means making infinitesimal changes to the network weights, there exists a neighborhood of $x_{0}$ such that $(f_{k})_{i}$ is linear on it. Besides, since $x_{0}\in\mathcal{O}$, again, even if it means making infinitesimal changes to the network weights, then $(f_{k})_{i}$ will not be trivial. Thus $\left\{x\mid f_{k}(x)_{i}=0\right\}\cap\mathcal{O}$ is
contained in an hyperplane with dimension $<d=\dim M$. ∎

Now we see that singular points occur on a (Lebesgue) measure zero set.

###### Theorem 3.6 .

Let the notation be as above.
Consider the distribution $\mathcal{D}$:

$$ $\mathbf{R}^{d}\ni x\mapsto\mathcal{D}_{x}=\mathrm{span}\{\nabla_{x}p_{i}(y|x,w),\,i=1,\dots c\}$ (13) $$

where $p$ is an empirical probability given by Sofmax and a score
function $S$ consisting of a sequence of linear layers and ReLU activations.
Then, its singular points
(i.e. points where $\mathcal{D}$ changes its rank)
are a closed null subset of $\mathbf{R}^{d}$ contained in the union of $\ell$
hypersurfaces, where $\ell$ is the number of layers.

###### Proof.

The
singular points of $\mathcal{D}$ coincide, by Lemma 3.4 with
the points where $J_{S}$ the Jacobian of $S$ as in (8 changes its
rank. By Lemma 3.5 this occurs in the union of $\ell$
hypersurfaces.
∎

We conclude by observing that, as a consequence of the proofs of Lemma 3.5
and Thm 3.6, the singular points of the distribution $\mathcal{D}$ are contained
in the non smooth points and such points are contained in a measure zero subset of the data space.
Hence, if we restrict ourselves to the open set complementing such measure zero set,
we can apply Frobenius Thm to $\mathcal{D}$ to get the learning foliation.

###### Remark 3.1 .

###### Lemma 3.4 .

###### Proof.

###### Observation 3.2 .

###### Lemma 3.5 .

###### Proof.

###### Theorem 3.6 .

###### Proof.

## 4 Experiments

We perform our experiments on the following datasets: MNIST[],
Fashion-MNIST[], KMNIST[] and EMNIST[], letters only, that we denote with Letters.
We also create a dataset that we call
CIFARMNIST: it is the CIFAR10 dataset[] cropped and transformed to
be $28\times 28$ gray-scale pictures.

Our neural network is similar to LeNet, with two convolutional layers, followed by a Maxpool and two linear layers with ReLU activation functions, see Fig. 5. This is slightly
more general than our hypotheses in Sec. 3.3.
The model is then trained on MNIST, reaching $98\%$ of accuracy.

Figure: Figure 5: The structure of our CNN - picture created with [].
Refer to caption: Images/nn_neurips2024.png

In Fig. 6 we compute the Jacobian of the network and we measure its rank
by looking at its singular values for 100 sample points in the data space $\mathbf{R}^{784}$ of the MNIST dataset (we
normalize the singular values by dividing them by the largest one).
The statistical significance of these experiments is detailed in Fig. 7.

Figure: (a) Singular values of $J_{N}$ at 100 data points in MNIST.
Refer to caption: x6.png

We see clearly that on points in the dataset
the singular values of the Jacobian $J_{N}$, are smaller, as we remarked in Obs. 3.2, as a consequence of
our Lemma 3.4.

Figure: Figure 7: DIM eigenvalues sorted by decreasing order evaluated on 250 points for each dataset.
Refer to caption: x8.png

We report DIM’s eigenvalues (logarithmic scale) for different datasets in Fig.
7. The vertical segment for each eigenvalue and each
dataset represents the values for 80% of the samples, while the colored area
represents the values falling in between first and third quartile. The horizontal
line represents the median and the triangle represents the mean.
The points in MNIST, the training
dataset, are clearly identifiable by looking at the colored area.

Figure: (a) Validation loss.
Refer to caption: x9.png

We perform a proof of concept knowledge transfer by retraining the
last linear layer of our model on different datasets.
We report in Table 2 the median of highest and lowest DIM eigenvalues in
logarithmic scale, their difference $\Delta$ and validation accuracies after
retraining of our CNN. We see a correspondence between
the median of the lowest eigenvalue and the validation accuracy, suggesting to explore
in future works the relation between DIM, foliations and knowledge transfer.

**Table 2: Parameters for Knowledge Transfer (logarithmic scale)**
| Dataset | Highest evalue | Lowest evalue | $\Delta$ | DIM Trace | Val. Acc. |
| --- | --- | --- | --- | --- | --- |
| MNIST | -1.78 | -8.58 | 6.70 | -1.52 | 98% |
| KMNIST | 0.49 | -7.75 | 7.76 | 0.37 | 75% |
| Letters | 0.11 | -7.99 | 7.82 | 0.48 | 80% |
| Fashion-MNIST | 0.14 | -8.08 | 7.76 | 0.12 | 81% |
| CIFARMNIST | 0.41 | -6.90 | 6.75 | 0.27 | 33% |
| Noise | 0.24 | -5.36 | 5.49 | 0.27 | NA |

## 5 Conclusions

We propose to complement the notion of learning manifolds
with the more general one of learning foliations. The
(integrable) distribution $\mathcal{D}$, defined
via the data information matrix (DIM), a generalization of the Fisher
information matrix to the space of data, indeed allows for the partition of the data space according to the leaves of
such a foliation via the Frobenius theorem.
Examples and experiments show a correlation of data points
with the leaves of the foliation:
moving according to the distribution $\mathcal{D}$, i.e.
along a leaf, the model gives a meaningful label, while moving in the orthogonal directions leads to greater and greater
classification errors.
The learning foliation is however both singular (drop in rank)
and non smooth. We prove that singular points are contained into a set
of measure zero,
hence making the learning foliation significant in the data space.
We show that points in the dataset the model was trained with have
lower DIM eigenvalues, so that the distribution $\mathcal{D}$
allows successfully to determine whether a sample of points belongs or
not to the dataset used for training. We make such explicit
comparison with similar datasets (i.e. MNIST versus FashionMNIST, KMNIST etc).
Then, we use the lowest eigenvalue of the DIM
to measure the distance between data sets.
We test our proposed distance by retraining our model on
datasets belonging to the same data spaces and checking the
validation accuracy. Our results are not quantitatively conclusive in this
regard, but show a great promise as a first step to
go beyond the manifold hypothesis and exploiting the theory
of singular foliations to perform dimensionality reduction and
knowledge transfer.

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 5
presentation: 2
questions: 1. Section 1 states that "at least in the cases most relevant for concrete applications, and calls for more sophisticated mathematical modeling". This is too vague to figure out the main point. What kind of concrete applications and challenges are considered here, and what sophisticated mathematical tool is used here for what challenge? More detailed discussions are required to develop this statement.

2. For singular points, it would be helpful to provide some intuitive examples or visualization results, and discuss more about the physical meanings and potential applications.

3. Is it possible to extend the results to other activation functions?

4. It would be better to apply this observation to the domain adaptation task, which considers how to effectively transfer knowledge between datasets.
soundness: 3
strengths: 1. It is interesting to apply the foliation structure for capturing geometric information of data.

2. The paper constructs a relation between the eigenvalues of the data information matrix and the ability of knowledge transfer between datasets.
summary: This paper proposes a manifold learning method based on the foliation structure. Particularly, the eigenvalues of the data information matrix are leveraged to measure the difference between two datasets, which reflects the degree of knowledge transfer from one dataset to another. Several benchmark datasets are used to evaluate the performance of knowledge transfer and its relation with the eigenvalues of the data information matrix.
weaknesses: 1. Section 1 states that "at least in the cases most relevant for concrete applications, and calls for more sophisticated mathematical modeling". This is too vague to figure out the main point. What kind of concrete applications and challenges are considered here, and what sophisticated mathematical tool is used here for what challenge? More detailed discussions are required to develop this statement.

2. It would be better to re-organize the structure of the submission, especially Section 1. Is it necessary to discuss machine learning and deep learning in Section 1? The connection between this part and the others is unclear. 

3. Section 2 is chaotic without a clear structure. In addition, this part just lists some existing works without further insightful discussions.

4. The formal definitions of some mathematical notations are missing. For example, for Eq. (3), it seems that the gradient vectors are used to construct a subspace, while the definition of "span" is missing.

5. The results regarding singular points and the measure are interesting while difficult to understand. It would be helpful to provide some intuitive examples or visualization results, and discuss more about the physical meanings and potential applications.

6. Is it possible to extend the results to other activation functions?

7. The experiment part is weak. Only several simple datasets are used. An interesting connection between DIM eigenvalue and the ability of knowledge transfer is constructed. Based on this, it would be better to apply this observation to the domain adaptation task, which considers how to effectively transfer knowledge between datasets.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. For Figure 4, what are the possible ways to define singular points with ReLU? Or the method does not work with networks with ReLU?

2. How does Figure 6 connect to the knowledge transfer experiments? What is the conclusion drawn from this visualization?
soundness: 3
strengths: 1. The foliage structure provides important analysis of datasets, for example, its tangent spaces. Also, the paper visualizes the structures and the eigenvalues comparisons well. 

2. The knowledge transfer experiment shows good results and can show the similarity between many datasets This could extend to large datasets and greatly decrease training time. The experiments of moving data in the direction of tangent space also provide a meaningful way to analyze datasets.

3. The paper provides a theoretical background to DIM and its relation with the established results, like the Frobenius theorem and Fisher information matrix.
summary: The paper introduces a novel data information metric (DIM) adapted from the Fisher information matrix. It then illustrates how to use DIM to visualize the structures of datasets and compute distances between datasets. The paper also explains data foliage structure, which splits data into submanifolds. The foliage structure is further separated into regular and singular points, which gives valuable insights into this dataset. The application is to apply knowledge transfer between *NIST datasets. It transfers knowledge of classification from one dataset to other datasets and obtains good accuracies.
weaknesses: 1. The experiments are limited to *NIST datasets, so it is not clear whether the knowledge transfer can be extended to more complex datasets and manifolds. For example, the classifier is only fine-tuned with the last layer, and there are no more comparisons. Thus, we cannot conclude whether DIM and knowledge transfer are correlated. If Table 2 has more explanations, that will help the readers to understand the conclusion the authors are trying to get.


2. The theoretical results made assumptions about the neural network architecture, so not applicable to other architectures. 

3. Some other papers compute distances between datasets. Reviewer recommends the authors to cite them as well:

Alvarez-Melis, David, and Nicolo Fusi. "Geometric dataset distances via optimal transport." Advances in Neural Information Processing Systems 33 (2020): 21428-21439. 

Hua, Xinru, et al. "Dynamic flows on curved space generated by labeled data." Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence. 2023.

---

confidence: 2
contribution: 3
initial_rating: 5
presentation: 1
questions: - Why is $D$ used for the dataset in line 113 and then immediately after, for the DIM?  
- What are the blue lines showing in Fig 4, and why are they irregularly spaced?  What is the significance of the coloring of points?
- “Hence, there is no foliation whose leaves fill the data space, naturally associated to it via Frobenius Theorem.” (L227)  --> The foliation does not fill data space, but rather the distribution defined by Eqn 4, correct?  What does the “naturally associated...” clause mean, and what does “it” refer to?
- In Figs. 2 and 3, isn’t the distribution as defined in L193 9-dimensional?  So is there an unambiguous sense of direction “moving tangentially to $\mathcal{D}$”?  What is the “same direction” (L212), just a randomly sampled direction in dataspace?
- What is necessary about integrability when making Figs 2 and 3?  Couldn’t similar directions be found in dataspace using the distribution definition (4) for a network with any arbitrary activation function?  What would change about Figs 2 and 3 for a non integrable distribution?  
- L306, are p and N(x) equivalent, and if so, why are they both included?
- “The points in MNIST, the training dataset, are clearly identifiable by looking at the colored area.” This seems to be one of the main experimental results.  The overlap between the distributions of eigenvalues suggests only a fairly large population of samples can be identified as the training data or not, right?  Additionally, the identifiability is all from the largest couple of eigenvalues, right?  Can you speak to the significance of this, when it’s the *lowest* eigenvalues that correlated best with validation accuracy in the knowledge transfer task (Table 2)?
- “We propose to complement the notion of data manifolds with the more general one of data foliations” -- how are data foliations more general than data manifolds?
- What is the value of Figs 5 and 8?  Why not show e.g. more traversals, perhaps with the starting point an image from a different dataset?
- Can you speak to computing the DIM in practice?  Could the proposed method easily be applied to an imagenet pre-trained ResNet?
- Is there anything noteworthy about the DIM eigenvalue spectrum on points that the model misclassifies?
soundness: 2
strengths: The combination of differentiable geometry and information geometry to study the data manifold is interesting, and the authors demonstrate that it can lead to some useful insights on real datasets.  In particular, the comparison of the eigenvalue spectrum of the DIM at data points from different datasets seems a reasonable and principled way to assess the model’s familiarity to different datasets.  The DIM seems like a useful construct, and should generally be more manageable to deal with than the FIM (data space being significantly smaller than parameter space); while it was introduced elsewhere, the authors demonstrate new utility from the DIM.
summary: The submission proposes an information geometry analysis of the data manifold.  In particular, with the “data information matrix” (DIM) -- analogous to the Fisher information matrix but where gradients are taken in data space -- the authors define a distribution over dataspace that can reveal aspects of the data manifold given a trained model.
With this distribution in hand, the authors empirically examine its nature as a foliation (or not, depending on the nonlinear activation in the network) and find that the points in the dataset are different from random points sampled over the dataspace.
From the ability to assess the rank of the distribution at different points in dataspace via the eigenvalues of the DIM, the authors compare the values at points from different datasets for a network trained on MNIST.  Finally, the MNIST-trained network is fine-tuned on the different datasets and the performance of the network is shown to correlate with the smallest non-zero eigenvalues.
weaknesses: The text is hard to follow at times.  Given the machine learning audience of ICLR, clarity of exposition about differentiable geometry concepts is important, yet explanations are often confusing or absent (see questions below).  I did not get anything from the visualization of Fig 4 (ReLU vs GeLU trained on XOR).  I am skeptical about the claim that points in the training data are identifiable -- perhaps en masse, but not individually.

---

confidence: 2
contribution: 2
initial_rating: 5
presentation: 2
questions: I think this paper is trying to contribute theoretically inspired methodology. Thus, would you mind answer the points I mentioned in the weakness?

Please carefully use \citet and \citep in your draft.
soundness: 3
strengths: ### Theoretical Foundation:
The paper provides rigorous mathematical foundation for understanding neural networks geometrically. Authors prove that singular points of the foliation structure lie in a measure-zero set, giving theoretical guarantees for their geometric framework. This connects deep learning to classical differential geometry in a novel way.

### Interpretable Framework:
The authors demonstrate clear visualization of how neural networks organize data through geometric structures. In their XOR example (Figure 4), they show how different nonlinearities (ReLU vs GeLU) create different foliation patterns, making abstract concepts concrete and visually interpretable.

### Quantitative Metrics:
The work introduces measurable quantities for dataset similarity through DIM eigenvalues. Their experiments show that MNIST transfer to Fashion-MNIST achieves 81% accuracy with λ(d-8) = -8.08, while transfer to CIFARMNIST only achieves 33% with λ(d-8) = -6.90, providing a quantitative relationship between geometric properties and transfer learning success.
summary: This paper presents a geometric analysis of neural networks through the lens of foliation theory. The authors introduced the Data Information Matrix (DIM), a variant of the Fisher Information Matrix applied to data space, to study how neural networks organize high-dimensional data. They developed theoretical proofs showing that neural networks create a foliation structure in the data space, where singular points exist in a measure-zero set. To validate their theory, they conducted experiments using a LeNet-like CNN architecture on MNIST-family datasets, demonstrating how DIM eigenvalues correlate with transfer learning success. They analyzed both ReLU and GeLU networks, showing different properties in terms of foliation structure and investigating the relationship between geometric properties and knowledge transfer capabilities. The experimental validation focused on small-scale problems with 28x28 grayscale images and primarily 10-class classification tasks.
weaknesses: ### Scale Limitations:
The framework faces severe computational challenges with modern-scale problems. For a typical 224x224x3 image, the DIM would require ~85GB memory, making it impractical for real applications. The experiments only deal with 28x28 grayscale images, avoiding these scaling challenges.

### Dataset Constraints:
The validation is limited to simple classification scenarios. While the theory claims generality, experiments only use MNIST-like datasets with 10 classes. This leaves open questions about handling CIFAR100 (100 classes) and ImageNet (1000 classes).

### Architectural Restrictions:
The theoretical analysis mainly focuses on ReLU networks with simple architectures. Modern networks using complex architectures like ResNet (or VGGNet) are not addressed. For example, their transfer learning experiments only retrain the last layer, which is far from contemporary transfer learning practices that often involve partial fine-tuning or adapter modules.

Meta-review: This paper proposes an information geometry analysis for the analysis of data manifold learning based on the notion of foliation structure. It proposes to consider “data information matrix” (DIM), a variation of the Fisher information matrix to detect a foliation structure on the data space. The eigenvalues of this data information matrix can be used to measure the difference between two datasets, which reflects the degree of knowledge transfer from one dataset to another. The paper provides theoretical results and an empirical evaluation.
 
Strengths:   
-combination of differentiable geometry and information geometry,   
-using foliation structure is novel in this context,    
-the theoretical framework is solid.  

Weaknesses:  
- the paper needs a restructuration,   
-some parts are difficult to understand and some precisions are needed,   
-experimental evaluation is weak.    

All the reviewers found that the paper has some merits but lacks enough clarity and a convincing experimental study.   
During rebuttal, authors have given answers to the remarks of the reviewers by providing additional explanations and precisions. But they did not report additional experimental results.  
During the discussion, reviewers recognized that the paper provides an interesting approach but they agreed on the fact that the contribution is not clear enough and the experiments are not convincing.  

As a consequence I propose rejection.  
However, this paper has an interesting basis and I encourage authors to improve their work for other venues.

Decision: Reject