Unveiling Transformer Perception by Exploring Input Manifolds
Authors: Alessandro Benfenati
Alfio Ferrara
Alessio Marta
Davide Riva
Elisabetta Rocchetti
Venue: NeurIPS 2025 poster ()
Keywords: input space exploration
geometric deep learning
transformers
Primary area: deep_learning
TL;DR: This paper introduces a general method for the exploration of equivalence classes in the input space of Transformer models.

Abstract: This paper introduces a general method for the exploration of equivalence classes in the input space of
Transformer models. The proposed approach is based on sound mathematical theory which describes the internal layers of a Transformer architecture as sequential deformations of the input manifold. Using eigendecomposition of the pullback of the distance metric defined on the output space through the Jacobian of the model, we are able to reconstruct equivalence classes in the input space and navigate across them. Our method enables two complementary exploration procedures: the first retrieves input instances that produce the same class probability distribution as the original instance—thus identifying elements within the same equivalence class—while the second discovers instances that yield a different class probability distribution, effectively navigating toward distinct equivalence classes. Finally, we demonstrate how the retrieved instances can be meaningfully interpreted by projecting their embeddings back into a human-readable format.

Unveiling Transformer Perception by Exploring Input Manifolds
Alessandro Benfenati lo Alfio Ferrara' 16 Alessio Martalc Davide Riva 2d
Elisabetta Rocchetti lb+
"Department of Environmental Science and Policy, "Department of Computer Science, Department of Mathematics, 4Department of Control and Computer Engineering 1 Università degli Studi di Milano, PPoliecnico di Torino (alessandro. benfenati, alfio.ferran feniara)ummmiii (alessio. marta, elisabetta. @lisabetta.rocchett] @uniiii.mmmmm...mmm {davide riva] 0polito..o........ it Corresponding author
@unimi. .it
Abstract
This paper introduces general method for the exploration of equivalence classes the input space of Transformer models. The proposed approach is based on sound mathematical theory which describes the internal layers of a Transformer architec- ture as sequential deformations of the input manifold. Using eigendecomposition of the pullback of the distance metric defined on the output space through the Jacobian of the model, we are able to reconstruct equivalence classes in the input space and navigate across them. Our method enables two complementary exploration procedures: the first retrieves input instances that produce the same class proba- bility distribution as the original instance-thi identifying elements within the same equivalence class-while the second discovers instances that yield different class probability distribution, effectively navigating toward distinct equivalence classes. Finally, demonstrate how the retrieved instances can be meaningfully interpreted by projecting their embeddings back into a human-reat -eaaalree format. Disclaimer: This paper includes examples of sensitive and very offensive language solely to illustrate the behavior of LLMs exploring the input space.
1 Introduction
In the literature, the investigation of the input space of Transformers relies on perturbations of input data using heuristic or gradient-based criteria [24] 17 14, on the analysis of specific properties of the embedding space (6] via the production of optimal robust explanations and counterfactuals. In this paper, we propose method for exploring the input space of Transformer models by identifying equivalence classes with respect to their predictions. Our approach based on sound mathematical theory which describes the internal layers of a Transformer architecture sequential deformations of the input manifold. Using eigendecomposition of the pullback of the distance metric defined on the output space through the Jacobian of the model, we are able to reconstruct equivalence classes in the input space and navigate in and across them. The equivalence class consists of the counterimage of a particular probability distribution: this means that the elements of equivalence class, once fed to the Transformer, will provide different realizations from the same probability distribution. Thanks to our approach, provide two different methods for exploring the embedding space: the Singular Metric Equivalence Class (SiMEC) and the Singular Metric Exploration (SiMExp) algorithms. The first allows for the identification of inputs within the same equivalence class. This means that, given two data inputs x, x'' identified through the exploration process as belonging to the same equivalence class and class label C, our method guarantees that the Transformers will assign the same probability, modulo numerical approximations, to that label for both inputs: p( p(Cll)) R (C'(a')
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

The second method, instead, allows for the exploration of the embedding space starting from an element within an equivalence class and moving towards a different equivalence class. This means that, given two inputs, namely n", identified in this way and class label C, we guarantee that the Transformer will assign different probabilities to that label for the two inputs, i.., pl (C(r)) * pCC'a"). Our experimental data show that this different probability assignment can lead to a change in the most probable class in the Transformers prediction.
In Section 2 we summarise the mathematical foundations of our approach. In Section 3 we present our method for the exploration of equivalence classes in the input of the Transformer models. In Section 4 we empirically investigate the effectiveness and applicability on Transformer models on textual and visual data. In Section [5] we discuss the relevant literature on embedding space exploration. Finally, in Section 6 we give our concluding remarks"
2 Preliminaries
We provide in this Section the theoretical foundation of the proposed approach, namely the Geometric Deep Learning framework based on Riemannian Geometry 2
A neural network is considered as a sequence of maps, the layers of the network, between manifolds, and the latter are the spaces where the input and the outputs the layers belong to.
Definition (Neural Network). A neural network is a sequence of C2 maps A, between manifolds of the form:
Mo
M1
Aa
M2
Aa
An
Mn-1
An Mn
(1)
We call Mo the imput manifold and M,, the output manifold. All the other manifolds of the sequence are called representation manifolds. The maps A; are the layers of the neural network. We denote by N(1) An ..... o Ai M, + Mn the mapping from the i-th representation layer to the output layer.
As an example, consider shallow network with just one layer, the composition of a linear operator A +6 with a sigmoid function a, where A E Rm* and E RT: then, the input manifold Mo and the output manifold Mj shall be R" and RT respectively, and the map (-(-) = o(A +b). We generalize this observation into the following definition.
Definition (Smooth layer). A map A; : M,-1 + M, is called a smooth layer if it is the restriction
to M a function X (x: Rd + R.d. of the form X (x) = A - bb²) .or i 1, ............. n, x E Rd, 500 R24, and A E IRd, xd: with F(i) Rdd. + Rd: a diffeomorphism.
Remark 1. Transformers implicitly apply for this framework, since their modules are smooth functions, such fully connected layers, GeLU and sigmoid activations, thus including also attention layers.
Our aim to transport the geometric information on the data lying the output manifold to the input manifold: this allows us to obtain insight on how the network "sees" the input space, how it manipulates it for reaching its final conclusion. For fulfilling this objective, we need several tools from differential geometry. The first key ingredient is the notion of singular Riemannian metric, which has the intuitive meaning of a degenerate sealar product which changes point to point - the starting point for defining a non-Euclidean pseudodistance between points of a manifold.
Definition 3 (Singular Riemannian metric). Let M R"
open subset of "." A singular Riemannian metric g over M is a map g : M > Ril x RR) that associates to each point p positive semidefinite symmetric bilinear form 9p R x R R in smooth way.
Without loss of generality, we can assume the following hypotheses on the sequence [ i) The manifolds M, are open and path-connected sets of dimension dim M, =4. ii) The maps A, are C submersions. iii) A; (M, = M, for every i = 1.... iv) The manifold M,, is equipped with the structure of Riemannian manifold, with metric g.. Definition 3 naturally leads to the definition of the pseudolength and of energy of curve.
'The code to reproduce our experiments can be found here Prausfoommrrseeeuullleleeee...

---

Definition 4 (Pseudolength and energy of a curve). Les y [a. - B curve defined on the (V. v) the pseudo- norm induced by the pseudo-metric 9p ar point p. Then the pseudolength of y and its energy are defined as
interval la, bj C T and ||e|| 9p(u,
PI(T) = J (1(())ll S.¹ (orca)(((()))
F(7) = 1" (1(()))
(2)
The notion of pseudolength leads naturally define the distance between two points.
Definition 5 (Pseudodistance)). Let z, y M = R" The psendodistance between and y is then Pd(x, y) (P(()) 7: [0, - M, r C² ((0, 1)), x(0) = ((1)=
(3)
One can observe that endowing the space R" with singular Riemannian metric leads to have non trivial curves whose length is zero. A straightforward consequence is that there are distinct points whose pseudodistance therefore zero: natural equivalence relation arises, iee. x ~ y +> Pd(æ, y) = 0, obtaining thus metric space (ER" ~ Pd).
The second crucial tool is the notion of pullback of function. Intuitively, given map M + N between two manifolds, the pullback operation allows to transfer the geometric information of the output N onto the input M by means of the Jacobian of f. More specifically, let f be function from Iº' to R", and fix the coordinate systems respectively. Moreover, we endow R" with the standard Euclidean metric g. whose associated matrix is the identity. The space Ip can then be equipped with the pullback metric f" whose representation matrix reads as:
(#1..... Tp) and y = (yi..... ..g) on R and on R",
2
afn Dx.
O Ox,
(f
Σ
ghkk
h,k=1
The sequence ① shows that a neural network can be considered simply as a function, a composition of maps: hence, taking f An An-1 .... Ai and supposing that Mo IR? M,,- IR9, the generalization of (4) applied to (I) provides with the pullback of a generic neural network.
Hereafter, consider in (T) the case M, R2, equipped with the trivial metric g( Ig. i.e.,
the identity. Each manifold M; of the sequence (T) is equipped with a Riemannian singular metric, denoted with g", obtained via the pullback of Nay)- The pseudolength of curve 7 the i-th manifold, namely P4(7), computed via the relative metric g' via
2.1 General results
We depict hereafter the theoretical bases of our approach. We denote with N₁ the submap o,o.. M; + Mn, and with A No the map describing the action of the complete network. The starting point is to consider the pair (Mr, Pd.): this a pseudometric space, which can be turned into a full-fledged metric space M./ ~ by the metric identification x ~i y (d,(x, v) O. The first result states that the length subsequent manifolds.
curve on the i-th manifold is preserved among the mapping on the
Proposition 1. Let [0, 1] + Mj be piecewise C curve. Let j E (0,i+) 1, ,n) and consider the curve 7 A5 ..... A, y on My. Then P4(7) P1,(7,)-
In particular this is true when A n. i.e., the length of a curve is preserved in the last manifold. This result leads naturally to claim that if two points are in the same class of equivalence, then they are mapped into the same point under the action of the neural network.
Proposition If two points p. 9 E M, are in the same class of equivalence, then N,(p) = NM(g)-
The next step is to prove that the sets M:/ me are actually smooth manifolds: to this aim, we introduce another equivalence relation: x ~N, y if and only if there exists a piecewise 7 [0, -> M, such that r(0) x, 7(1) = and N₁ e (s) (2xx)) Vs E j0, 1]. The introduction of this equivalence relation allows us to easily state the following proposition.
Proposition 3. Let x, y E M,, then z ~ y if and only if ~M,, y.
The following corollary contains the natural consequences of the previous result; the second point of the claim below is the counterpart of Proposition] 2
3

---

Corollary 1. Under the hypothesis of Proposition 3 one has that M,/~; -M./~N. Moreover, if two points p. q E Mi are connected by a a curve 7 [0, M, satisfying N:(p) =N 7()) for every 8 E [0, 1), then they lie in the same class of equivalence.
One is then able to prove that the set M1/ ~y is a smooth manifold:
Mi is smooth manifold of dimension dim())(Mo)).
Proposition 4. ~
This last achievement provides practical insights about the projection x, on the quotient space, that consists in the building block of the algorithms used for recovering and exploring the foliation in equivalence classes neural network.
Proposition 5. TH M, + M1/ me is a smooth fiber bundle, with Ker(dx,) = VM, which is therefore integrable distribution. VM; is the vertical bundle Mj. Every class of equivalence [p] is path-connected submanifold of M; and coincide with the fiber of the bundle over the point p E M,
In 3 it is shown that these results keep to hold true in the case of convolutional layers and residual connections.
Methodology
The results depicted in Section 2.1
provide powerful tools for investigating how a neural network sees the input space starting from point x. In particular we point out the following remarks: i) If two points y belonging to the input manifold Mo are such that x ~v then N(xx) Ny)): ii) given point = E Mn, the counterimage (z) is a smooth manifold, whose connected components are classes of equivalence in Mo with respect to ~vo, then necessary condition for two points x, y E Mo to be the same class equivalence is that M(x) = Ny)); iii any class of equivalence [x]]. x Mo. is maximal integral submanifold of V.M. The above observations directly provide with a strategy to build up the equivalence class of an input point x E My- Proposition tells us that VM, is an integrable distribution, with dimension equal the dimension of the kernel of gº: can hence find dim(K (er(g")) vector fields which are a base for the tangent space of My. This means that we can compute the cigenvalue decomposition of g and consider the L linearly independent cigenvectors, namely ( associated to the null eigenvalue: these eigenvectors depend smoothly on the point, fact that is not trivial when the matrix associated to the metric depends several parameters [1]. We can build then all the null curves by randomly selecting one eigenvector E {wi and then reconstruct the curve along the direction ū from the starting point æ. From practical point of view, one is led to solve the Cauchy problem with first order differential equation + = ū and initial condition (0) A.
Algorithm 1 The Singular Metric Equivalence Class (SiMEC) algorithm.
Algorithm The Singular Metric Exploration (SiM- Exp) algorithm.
1: Set the network N; choose the number of itera- tions K. Choose the input x:
1: Set the network N: choose the number of itera- tions K. Choose the input 2 (0)
2: for l 0, 1,..,... 1 do
2: for 0, 1,..,, - do
3: Compute ((()))))
Compute (x(,)())
4: Compute the pullback metric 20m
4: Compute the pullback metric g
5:
Diagonalize g.(e) and find the eigenvectors
5: Diagonalize gh and find the cigenvectors
{ux he associated to the zero eigenvalue
{wi heL, associated eigenvalues λ Randomly select E (w) Jwe)
6: Randomly select E {m Ie
7:
g(() 77 minj:,, /Ail/ maxy PAyl
7:
g(k) T/ min /Ayl/ maxy Pyyl
8:
r (k+1) +- (k) + (4))
8: (k+1) +- (k) + 8
9: end for
end for
Project (k++1) the feasible region X
10: Project y (k+1) to the feasible region X'

---

3.1 Input Space Exploration
This entire procedure is coded in the Singular Metric Equivalence Class (SiMEC) and Singular Metric Exploration (SiMExp) algorithms, whose general schemes are depicted in Algorithms [ and[] SiMEC reconstructs the class of equivalence of the input via exploration of the input space by randomly selecting one of the eigenvectors related the zero eigenvalue. On the opposite, in SiMExp, in order to move from class of equivalence to another we consider the eigenvectors relative to the nonzero eigenvalues. This requires the slight difference in lines 5-6 between Algorithm Land Algorithm 2
There are some remarks to point out. From numerical point of view, the diagonalization of the pullback may lead to have even negative eigenvalues: hence one may use the notion of energy of a curve, related to the pseudolength. the values g(k) are too small more iterations are needed to move away from the starting point sensibly. Therefore there is a trade-off between the reliability of the solution and the exploration pace. Relying on the theory of dynamical systems, we can in practice estimate ((A) at each iteration with the inverse of the square root of the condition number T = max; /xyl/ xo AAyl of the pullback metric 9((s), as in a locally-linearized dynamical system. This is our default choice for both algorithms. We multiply the default value by multiplier 7) in order to explore the sensitivity of Algorithm| [ and Algorithm 2 to variations of step length, Indeed, to build points in the same equivalence class Algorithm| I needs to follow null curve closely with as little approximation as possible. In contrast Algorithm 2 whose goal is to change the equivalence class, does not have the same problem and larger ö are allowed.
expecting Algorithm|| to be more sensitive compared to Algorithm 2
In the final step of each iteration of both algorithms, the embeddings x; ...x(()))) need to be con- strained a feasible region X. This region is defined by the distribution of embeddings derived from the embedding layer, which is bounded by definition. Specifically its upper bound has components UB, E0 Esg (ax + +ZJE Esg min + max, (g(s) where E is the embed- ding layer weight matrix, (xy) represent input features, bounded 0 and in both the visual and textual case, and q(s) is the positional encoding vector at position 8 in the sequence of patches/tokens. The lower bound LB is obtained by switching max with min and vice versa. We acknowledge wldge that these bounds present non-null margin from the actual embedding distribution domain, however we find them to be a suitable estimate for the practical purpose of interpretation of input space exploration results. presented in next section. Notwithstanding numerical approximation errors, the outputs of SiMEC algorithm at each iteration k are predicted by '' to yield the same probability distribution [p(-Iz:0))) - p(- (=(()) < E. O < E < 1, iee the original input probability, by construction. On the contrary, SiMExp algorithm induces a non-null probability change in the output space, which might possibly lead to a class ranking change, i.e. the situation where class A is given higher probability than class B at iteration k: - but it is surpassed by class B iteration k, and eventually prediction change, i.e. the situation in which ar argmar (N (2x:: (R)) v/ # argmax (N (x))))) for le in non-degenerate neighborhood of a Mhange-point iteration T.
As for the computational complexity of the two algorithms, the most demanding step is the computa- tion of the eigenvalues and eigenvectors, which is O(n²), with n the dimension of the square matrix g gg 20). Since all the other operations are either O(n) or On?)), we conclude that the complexity
of both Algorithms
and 2 is O(n)).
3.2 Interpretation
Algorithms and a for the exploration of the equivalence classes in the input space of Trans- former model. However, the points explored by these algorithms may not be directly interpretable by human perspective. For instance, image "readable" by a human observer. Here we present an interpretation method for Transformers based on input space exploration, which is then demonstrated on two Vision Transformer (ViT) models trained for image classification [8), and two BERT models, one trained for masked language modeling (MLM) 7 and the other fine-tuned for text classification [18]
piece of text may need be decoded to be
Using SiMEC and SiMExp to explore the embedding space reveals how Transformer models perceive equivalence among different data points. Specifically, these methodologies facilitate the sequential acquisition of embedding matrices 2(()) (K), as detailed in Algorithms and 2 A key feature
5

---

of the SiMEC/SiMEXp approach is its ability to selectively update specific tokens (for text inputs) or patches (for image imputs) during each iteration. This selective update allows explore targeted modifications that prompt the model to either categorize different inputs as the same class or recognize them as distinct. Unlike other approaches [24 [2] where perturbations are predetermined, this method lets the model itself guide us to understand which data points belong to specific equivalence classes.
To interpret embeddings produced by the exploration process, they must be mapped back into human-understandabhl form, such as text or images. The interpretation of an embedding vector depends on the operations performed by the Transformer's s embedding module Σr. &r consists only of invertible operations, it is feasible to construct layer that performs the inverse operation relative to Eg. The output can then be visualized and directly interpreted by humans, allowing for a comparison with the original input to discern how differences in embeddings reflect differences in their representations (e.g.. text, images). If the operations in Er are non-invertible, a trained decoder required to reconstruct an interpretable output from each embedding matrix x(1)) ,.... z((k), Such operation injects some unforeseen noise into the interpretation results, which is investigated in the experimental setting. In practice, we construct the ViT models such that the embedding layer is invertible, whereas for BERT models it is feasible to exploit layers that are specialized for the MLM task to map input embeddings back tokens. This approach is effective whether the BERT model in question is specifically designed for MLM or for sentence classification. In the case of sentence classification models, necessary to select corresponding MI. BERT model that shares the same internal architecture, including the number of layers and embedding size.
Algorithm depicts the process of interpreting SiMEC/SiMExp outputs for both VIT and BERT ex- periments. After initializing the decoder according the model type, the embeddings z! z0)),... .... (KK) are decoded and the selected segments for exploration are extracted. These segments are then used to replace the corresponding parts of the original input instance.
Algorithm 3 Interpretation for Exploration results for ViT and BERT models.
1: Inputs:
2: Transformer model 7 with: Patcher/Tokenizer Tr. Embedding layer &r. Input image/text
3: Modified embeddings z(() ..... resulted from Algorithm | or [ applied on r!() = (Tr((()))
4: P " (1, [ indices of patches/tokens to update
5: If is VIT:
6: Initialize decoder d with weights from Eq.
7: I is BERT:
Initialize decoder with intermediate and final layers of a BERT for MLM task. 9: Decode modified embeddings (0)) !!) using d generate the images/sentences Tb... Z'x.
8:
10: For each z' E Z': replace segments relative to indices P in z with those in z'.
11: Outputs:
12: Modified input images/sentences, for each SiMEC/SIMExp iteration.
Figure I (top left) presents the outcome of applying Algorithm 3 to ViT exploration experiment on a CIFAR10 image. Both SiMEC and SiMExp produce visually similar outputs- each still resembling "cat" to human observer yet the SiMExp interpretation is classified "dog" iteration 750. This demonstrates how subtle modifications, such as changes in background pixels, can significantly influence model predictions, even when such changes perceived irrelevant by humans, as also noted in [24] A clear difference emerges the exploration dynamics of the two algorithms (Figure 1. top right): SiMExp progresses in more straight and directed manner, reflecting its goal of escaping the initial equivalence class. This divergence is further illustrated in the bottom right subplots, where the class probability distributions remain stable during SiMEC exploration but show notable fluctuations under SiMExp exploration. The lower part of Figure I shows a similar example for textual data from the Seasuring-Hate-Speeer (MHS) dataset. In this case, SiMExp identifies an alternative sentence that is classified as "Hatespeech", contrasting with the original input, which had been classified as "Offensive".
Experiments
Experiments are conducted on textual and visual data and are aimed at two objectives: (i) obtaining an empirical verification of the behavior of SiMEC and SiMExp under diverse settings, and (ii) verifying

---

20 5VD Whssskal on of SSeec sericg daakligg
Hepoltoon apsl mng. (C1 settif orp sndl pucchic dct reulloiic 2010
0.038
Clarr Pww.ehalliee..oooo
SMEXp-780
200
100
200500 in of SIMC Saitiig Ertwedings
Original
Repedtoon of lipu earranne, 3110 (49-5) euplore with all isleen and dets mudrplee 10.C
Cardo -etinieeeteeeee pilo
holopedd trdeoveelddd
sertug
comprce ye1
SWED--26 260
0.35
8.2
SVD Compurert
ronen ifherniin haseepeech SVEP
[CUR)
0.2
200
100
fasadon
Figure 1: (Top figure) Example of exploration on a CIFARIO image using SiMEC and SiMExp. Left: Original image, followed by interpretation outputs of 3750 from SiMEC (middle) and SiMExp (bottom). Right top: SVD projection of the explored points ,.!),.... x(() for SiMEC (red) and SiMExp (blue), where color intensity encodes iteration progress (darker colors correspond to later iterations), and point shapes indicate predicted class labels. Right bottom: Evolution of class probabilities over iterations, for SiMEC (left) and SiMExp (right). (Bottom figure) Example of exploration on an MHS sentence using SiMEC and SiMExp. Visualization layout and interpretation are analogous to the top figure.
the consistency of interpretation outputs with the ones from exploration only, in order to test their usability as alternative input data.
In each data modality, we experiment with two datasets presenting different features: (i) MNIST 13
a grayscale digit image dataset: (ii) CIFARIO I.. RGB object image dataset; (iii) WinoBias 25
textual dataset for MLM, especially focused on gender bias; (iv) Measuring- Seaasurng-HHate-peeee Spee-S (MHS) [16. a textual dataset for Text Classification, especially focused on hate speech detection. We trained one ViT model for each image dataset, and we used pretrained BERT models for MHS and Wino- Bias. More details about adopted models, experimental results further configurations, and full experimental details are provided in the Supplementary Materials.
Input space exploration For objective (i) we consider the following metrics: effectiveness of the exploration can be measured by changes in prediction probabilities as well as estimation of the hyper-volume explored, while speed is assessed in terms of total time (in seconds Algorithms are run for K = 1000 iterations, which we prove sufficient to capture their behavior, with delta multiplier 7 E {1; 10), the latter used with the aim to verify whether it is possible to speed up the process pace without compromising its stability. Finally, the experiments reported here all refer to the configuration in which all patches of an image are modified, while for textual inputs only the token with the highest attribution value is subject to exploration.
Given the predictions obtained by re-applying the models' encoder and classifier layers to the modified embeddings x² at each iteration k, we observe the changes in class probabilities. The theoretical
2 All experiments are based on the current PyTorch implementation of the algorithms and run on Ubuntu 22. 04 machine endowed with one NVIDIA H100 GPU and CUDA 12.4.
7

---

results suggest that SiMEC should induce minimal fluctuations in them, while SiMExp should yield rapidly changing probabilities, up to prediction changes,
Original and top classes' probabiliy from embeddinng across datasets
CIFARIO
MNUST
MIS
Windfllx
%
(a) SiMEC and SiMExp
Winobess
(b) Baseline gradient-base based Gaussian noise approaches
Figure 2: Mean and standard deviation (where applicable) of probability values for the original class (solid line) and the top predicted class (dashed line) based on embeddings obtained during exploration, across iterations and datasets. Subfigure (a) depicts the behavior of SiMEC (orange) and SiMExp (blue), while subfigure (b) reports the behavior of corresponding baseline algorithms. SiMExp results in notable decrease in the probability of the original class, while the probability of the nighest-scoring class decreases to a lesser extent, indicating a shift in the most probable class.
Figure 2a depicts the empirical reflection of the theoretical results. Focusing the probability of the original input class (i.e. class predicted at k = 0), we see that SiMEC manages to keep it constant while SiMExp makes it drop significantly within the first 1000 iterations. As baseline, we compare our results with a gradient-based Gaussian-noise approach which updates the input embedding (k) at each iteration by (zV)( + e, where the sign is determined by what exploration we are performing (same class vs other class) and is a small Gaussian noise vector orthogonal to the gradient to guarantee exploration. Applying the same number of iterations and the same average step size 8 in each experiment allows us to conclude that SiMEC and SiMExp are significantly more effective than the baselines for staying in the original equivalence class and moving to another class respectively. Indeed, in SIMEC original class probability always follows strictly the top class probability, which in the baseline is rarely the case; in SiMExp the two probabilities tend to diverge as predicted by the theoretical analysis, while in the baseline they remain close to one another.
In order to verify the actual shift in class probabilities, we report in Table the statistics about average class ranking changes, which indicate a clear tendency of SiMExp changing equivalence classes compared to SiMEC
Furthermore, we estimate the per-patch explored hypervolume by reducing embedding vectors to the first principal components, retaining 90% of the total variance, and computing at each iteration & the clement-wise difference A (R) (max:=o..... (t) min (0) ) (the 1/n
power 1/n allows for more stable computation). The product of the components of (k) gives
the volume of an hyper-dimensione cuboid which contains the explored region and is thus an over-estimate of the scope achieved by the exploration. By computing the average volume ratio (K)
PV (ILA (K) SiMd Ezp/ / /11A SiM )C)". we empirically verified that SiMExp explores a portion of space that is bigger than the one explored by SiMEC by an order of 10a. We validated these results by performing Welch t-tests on py: all p-values resulted lower than 10-3 Thus we conjecture that the exploration took a privileged direction on SiMExp experiments, thus making the volumes increasing faster than in SiMEC experiments.
8

---

MNIST
CIFARIO
WinoBias 0.0
MHS
SiMEC
1] =1
0.03 (0.18)
0.0
0.0
10
0.07 (0.25)
0.12 (0.33)
0.0
0.0
SiMEXp
1=1
1.31 (1.64)
2.60 (3.35) 8*
(0.33)
0.75
(0.89)+
77 10 11.91 (11.87)+
18.70 (15-90)* **
3.25
(2.41)
3.64 (3.43) *
Table I: Average class ranking changes per 1000 iterations across datasets and values. The class ranking change is computed by counting pairwise inversions in class rankings before and after each exploration update. Results are reported as mean (standard deviation) over multiple runs and inputs. The symbol ** indicates that these ranking changes involve at least once a prediction change, i.e., change in the top predicted class. SiMExp induces substantially more ranking and prediction changes than SiMEC, especially for larger 77.
Finally, we measure the time required to explore the input space of a model with the SiMEC and SiMExp algorithms. Means (and standard deviations) of required times are computed per patch/token and per iteration: 0.126 s (0.008) for CIFARIO, 0.050 (0.004) for MNIST, 0.300 s (0.020) for Winobias, and 0.310 (0.074) for MHS.
Using interpretation outputs as alternative input data Objective (ii) is to assess whether our interpretation algorithm (Algorithm 3) can generate alternative input data that either preserve the original input's equivalence class or shift to a different one, depending on the exploration dynamics of SiMEC and SiMExp. The mean difference of pixel/tokens generated from iteration to iteration amounts at 2.219. 10 for SiMEC experiments, and at 83.028 10 for SiMExp experiments; these values increase as the number of explored patches. These results show that our algorithms generate diverse outputs across iterations, especially when the exploration is performed following the SiMExp algorithm.
Beyond output diversity, we evaluate whether and how the model's prediction for the original equivalence class evolves as SiMEC and SiMExp explore the embedding space. Specifically, we track how the class probabilities from the modified embeddings change when decoded back into the data domain every k iterations. This evaluation differs from simply observing changes in embedding predictions. The projection step (Algorithms and 2 sphere containing the data domain, not the exact input space. Additionally, the decoder itself introduces approximations, either due to model limitations or numerical errors. These factors can cause discrepancies between predictions from embeddings and from decoded interpretations. To quantify this effect, we compute the average Wasserstein distance (p 1) between probability distributions predicted from embeddings and their corresponding interpretation outputs. Wasserstein distance is preferred over KI. divergence for its interpretability in this context. Results show that, with median Wasserstein distance of 0.0, SiMEC produces interpretation outputs whose predicted probabilities remain very close to those of the embeddings, indicating consistent generation of reliable alternative input data. In contrast, SiMExp's 's outputs exhibit larger and more variable Wasserstein distances, whose distribution has a median of 0.049, highlighting inconsistencies between embeddings and their interpretations.
step 11) constrains modifications to an d
In cases where predictions on SiMExp's interpretations initially differ from prediction on SiMExp's s embeddings, an average of 10.9% of these misaligned predictions eventually realign exploration progresses. On average, this "catch-up" effect occurs after an average of 290.65 iterations, suggest- ing that longer exploration trajectories (beyond 1000 iterations) could further improve alignment. Supporting this observation, we find an average positive Pearson correlation of 0.32 (average p- value 0.08) between top class' probability predicted on SiMExp's 's embeddings and its corresponding probability prediction on SiMExp's interpretations, indicating trend towards convergence.
5 Related work
Works dealing with embedding space exploration mostly focus on the study of specific properties of the embedding space of Transformers, especially NLP. For instance, Cai et al. [6) challenge the idea that the embedding space is inherently anisotropic 9 discovering local isotropy, and find
low-dimensional manifold structures in the embedding space of GPT and BERT. Biś et al. [4] argue that the anisotropy of the embedding space derives from embeddings shifting in common directions

---

during training. In the field of Computer Vision, Vilas et al. 23 map internal representations of a VIT onto the output class manifold, enabling the early identification of class-related patches and the computation of saliency maps on the input image for each layer and head. Applying Singular Value Decomposition to the Jacobian matrix of ViT, Salman et al. [77] treat the input space as the union of two subspaces: one in which image embedding doesn't 't change, and another one for which it changes. Except for the last one, all the aforementioned approaches rely on data samples. By studying the inverse image of the model, instead, we can do away with data samples. The idea of applying Riemannian geometry to capture geometric information about the input manifold of a neural network building foliation in equivalence classes has also been explored in [10] 21 22) in the case of simple architectures. In these works a foliation of the data domain is obtained by means of the pullback of variation of the Fisher information matrix for classifier networks with ReLU and softmax activation functions, with applications to knowledge transfer and the study of adversarial attacks.
In contrast to these works, we apply Riemannian geometry techniques to study the embedding space of transformers, computing the pullback of the metric of the output space, and we address the further problem of interpreting the output of the exploration process. Furthermore, our algorithms explore the embedding space dynamically, with a non-fixed choice of the integration step 6.
6 Conclusions
Our exploration of the Transformer architecture through theoretical framework grounded in Rie- mannian geometry led to the application of our two algorithms, SiMEC and SiMExp, for examining equivalence classes in the Transformers' input space. In particular, our method enables two com- plementary exploration strategies, one for retrieving input instances that produce the same class probability distribution different class probability distribution. We demonstrated how the results of these exploration methods be interpreted in humablreadable form and how the exploration outputs can be used to generate alternative input data.
the original instance, the other for discovering instances that yield
Future research directions include delving deeper into the potential of our framework for controlled input generation within an equivalence class. Our goal is to investigate how, in the XAI scenario, framework can facilitate local and task-agnostic explainability methods applicable to Computer Vision (CV) and Natural Language Processing (NLP) tasks, among others. In particular, we see methods potential approaches to investigate Transformers' sensitivity and explainability with respect to input data features. In futuree applications to large-scale architectures where the dimension of the embedding space is of order 103 - 10 19), we also plan improve the scalability of the SiMEC and SiMExp algorithms, e.g. making use of partial decompositions.
Acknowledgments and Disclosure of Funding
This work is partially supported by PNRR-NGEU program under MUR 118/2023.
References
[1] A. Benfenati and A. Marta. A singular Riemannian geometry approach to Deep Neural Networks
I. Theoretical foundations. Neural Networks, 158:331-343 2023.
[2] A. Benfenati and A. Marta. A singular riemannian geometry approach to deep neural networks
ii reconstruction of 1-d equivalence classes. Neural Networks, 158:344-355 2023.
[3] A. Benfenati and A. Marta. A singular riemannian geometry approach to deep neural networks ii.. piecewise differentiable layers and random walks on n--iiennional classes, 2024. URL https://aaxixs org/ hat.b///22444466. 06104
[4] D. Bis, M. Podkorytov, and X. Liu. Too much common: Shifting of embeddings in trans- former language models and its implications. In Proceedings of the 2021 conference of the North American chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5117-5130, 2021.
10

---

[5] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in neural information processing systems,
[6] X. Cai, J. Huang, Y. Bian, and K. Church. Isotropy in the contextual embedding space: Clusters and manifolds. In International conference on learning representations, 2020.
[7] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In J. Burstein, C. Doran, and T. Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Volume I (Long and Short Pa- pers), pages 1171-4111 1776,416, Minneapolis, Minnesota, June 2019, Association for Computational Linguistics. doi: 10.18653/v//N19) 1423.
[8] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenbom, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale, 2021. arXiv:2010.11929.
[9] J. Gao, D. He, X. Tan, T. Qin, L. Wang, and T.-Y. Liu. Representation degeneration prob- lem in training natural language generation models. In International conference on learning representations. volume abs/1907 12009, 2019.
[10] L. Grementieri and R. Fioresi. Model-centric data manifold: The data through the eyes of the model. SIAM Journal on Imaging Sciences, 15(3):1 140-1156, 2022. doi: 10.1137/21M1437000
[11] A. Krizhevsky, G. Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[12] E. La Malfa, R. Michelmore, A. M. Zbrzezny, N. Paoletti, and M. Kwiatkowska. On guar- anteed optimal robust explanations for nlp models. In Z.-H. Zhou, editor, Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21, pages 2658- 2665. International Joint Conferences on Artificial Intelligence Organization, 2021. doi: 10.24 0..24963/jcai.222213333
[13] Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied document recognition. Proceedings of the IEEE, 86(1 11:22)8-22233 22224, 1998. doi: 10. 109/5.726791
[14] N. Papernot, P. McDamiel, S. Jha, M. Fredrikson, Z. B. Celik, and A. Swami. The Limitations of Deep Learning in Adversarial Settings. In 2016 IEEE European Symposium on Security and Privacy (EuroS&P), pages 372-387, Mar. 2016. doi: 10. 10.1100/000SP..20....6
[15] F. Rellich and J. Berkowitz. Perturbation Theory of Eigenvalue Problems. New York University. Institute of Mathematical Sciences. Gordon and Breach, 1969. ISBN 9780677006802.
[16] P. Sachdeva, R. Barreto, G. Bacon, A. Sahn, C. von Vacano, and C. Kennedy. The measuring hate speech corpus:
Leveraging rasch measurement theory for data perspectivism. In G. Abercrombie, V. Basile, S. Tonelli, V. Rieser, and A. Uma, editors, Proceedings of the 1st Workshop on Perspectivist Approaches NLP @ I.REC2022, pages 83-94, Marseille, France, June 2022. European Language Resources Association.
[17] S. Salman, M. M. B. Shams, and X. Liu. Intriguing equivalence structures of the embedding space of vision transformers, 2024.
[18] C. Toraman, F. Şahinuç, and E. H. Yilmaz. Large-scale hate speech detection with cross- domain transfer. In Proceedings of the Language Resources and Evaluation Conference, pages 2215-2225, Marseille, France, June 2022. European Language Resources Association.
[19] H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi. Y. Babaei, N. Bashlykov, S. Batra, Bhargava, Bhosale,
Bikel, L. Blecher, C. C. Ferrer, M. Chen, Cucurull, D. Esiobu, J. Fernandes, J. Fu, W. Fu, B. Fuller, C. Gao, V. Goswami, N. Goyal, A. Hartshom, S. Hosseini, R. Hou, H. Inan, M. Kardas, V. Kerkez, M. Khabsa, Kloumann, A. Korenev, P. Koura, M.-A. Lachaux, T. Lavril, J. Lee, D. Liskovich, Y. Lu, Y. Mao, X. Martinet, T. Mihaylov, Mishra, Molybog, Y. Níe, A. Poulton, J. Reizenstein, R. Rungta, K. Saladi, A. Schelten, R. Silva, E. M. Smith, R. Subramanian, X. E. Tan, B. Tang, R. Taylor, A. Williams, J. X. Kuan,
11

---

P. Xu, Z. Yan, I. Zarov, Y. Zhang, A. Fan, M. Kambadur, S. Narang, A. Rodriguez, R. Stojnic, Edunov, and Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023, URL https://arxiv org/ abs .92888
[20] L. N. Trefethen and D. I. Bau. Numerical linear algebra. Twenty-fithh anniversary edition, volume 181 of Other Titles Appl. Math. Philadelphia, PA: Society for Industrial and Applied Mathematics (SIAM), 2022. ISBN 978-1-61197-715.5
[21] E. Tron and E. Fioresi. Manifold learning via foliations and knowledge transfer, 2024. arXiv:240900441..
[22] E. Tron, N. Couëllan, and S. Puechmorel. Adversarial attacks on neural networks through canonical riemannian foliations. Machine Learning, 113(11-12:8655-8866 Oct. 2024. ISSN 1573-0565. doi: 10. 10007//11999444224406644444
[23] M. G. Vilas, T. Schaumlöffel, and G. Roig. Analyzing vision transformers for image classi- fication in class embedding space. Advances Neural Information Processing Systems, 36, 2024.
[24] M. Wu, H. Wu, and C. Barrett. Verix: Towards verified explainability of deep neural networks. Advances neural information processing systems, 36, 2024.
[25] J. Zhao, T. Wang, M. Yatskar, V. Ordonez, and K.-W. Chang. Gender bias in coreference resolution: Evaluation and debiasing methods. 2018.
12

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: In the abstract and introduction we claim that we present method for the exploration of equivalence classes in the input space of Transformer models, which is
analyzed depth Section 3 The mathematical theory we refer is deepened in Section
2
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings. is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Although our paper doesn't present a standalone Limitations section, discuss the limitations and tradeoffs given by numeric integration in Subsection 3.1 and theoretical assumptions are enumerated in Section [ Computational efficiency our algorithms is discussed in Subsection 3.1 Although we conducted experiments on datasets, these are representative of large variety of scenarios in the fields of NI.P and CV.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "I "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
13

---

3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: The full proofs part of two previously published papers which we cannot disclose for anonymity requirements. We replicate the relevant proofs in the supplementary material, part of which will be removed from the final version of the paper, referencing instead to the other papers.
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Pseudo-code of the proposed algorithms is reported in Subsections 3.1 and 3.2 to make the algorithms reproducible, plus our implementation is made available in the supplementary material. Experiments, including the complete setting, the hardware and the evaluation metrics are described in Section 4
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided instructions for how to replicate the results, access to a hosted model (e.g.. in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
detailed
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
14

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: All experiments are made reproducible through scripts provided as supplemen- tary material. The datasets used in the experimental setting all publicly available and widely known datasets.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ /nips.ccc/pullc//guu.
Cooe/ubmeeeeeeeee wwailbmmmsssinnni ssionPolicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if' applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All details provided in Section details of the analyzed architectures, number of iterations and values of the hyperparameter r of the SiMEC/SiMExp algorithms, technical infrastructure which the experiments were performed, amount of data the experiments were performed on.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
15

---

The standard deviation is reported for all experiments that support the claims of the paper, including the supplementary material. When not reported explicitly tables, where it is written in brackets, it is drawn on charts as shaded area.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula,
call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: All the experiments were performed on the same infrastructure, which is reported in footnote in Section Time of execution is of the key indicators reported for the Input space exploration experiments. More computing power would be required for experiments on bigger Transformer models.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the http:p::://eerpppssccc/u//////ta htc.:publc/tthiccuuleee........e NeurlPS Code of Ethics https: pttttps:/neuurigge
Answer: [Yes]
Justification: We comply with the terms of use the datasets employed in the experiments, and deem work has potentially harmful effect people safety, security, discrimi- nation, surveillance, harassment, nor on human rights. Our proposal does not contribute to spread bias and unfaireess towards certain groups of people nor to harm the environment.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
16

---

If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [No]
Justification: Although the impacts of XAI on society is broad and deep, in this paper we focus only on the technical problem of exploring the equivalence classes in the input space of Transformers. We conjecture that solutions this problem do not have direct societal impact, which is more relevant in the scope of applications using our solutions as tool for XAI and/or sensitivity analysis, as we discuss in the conclusions.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended
(e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymentt of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI...
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (c.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
17

---

12. Licenses for existing assets
Question:
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The datasets used in the paper are explicitely mentioned in the references, as required by their terms use.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of in the package should be provided. For popular datasets, paperswi wwwode....d...... com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Our work doesn't include crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
18

---

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: Our work does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
19


Reviews:
[{"review_id": "hThHHDd9gD", "reviewer": "Reviewer_bq1s", "summary": "This work develops tools using differential geometry to understand how changes to the input manifold translates to changes in the output. manifold. In particular, the authors interpret a neural network as a sequence of transformations and divide the inputs into equivalence classes depending on if they produce the same output. The authors present two algorithms to move across equivalence classes or stay within the same equivalence class, which they use to analyze networks trained on different text and vision tasks.", "questions": "1. I found the SimEXP algorithm to be remarkably similar to signed-gradient techniques for adversarial samples. Can the authors comment on the similarities or differences to literature on adversarial samples? Why is it surprising in this context that SimEXP works?\n2. In similar spirity, one would expect small changes to the input to not change the outputs. Isn't it unsurprising that SimEC is able to find very similar looking images that make the same predictions? \n3. What is the distance traveled in weight space for SimEC and SimExp. Is the distance traveled by SimEC extremely small, in which case aren't the results to be expected?", "limitations": "The main limitations are addressed in the paper.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 4, "significance": 2, "originality": 1, "strengths_and_weaknesses": "**Strengths.** \nThe authors do a great job of presenting the content, and provide an interpretation of neural networks as a series of deformable transformations. While the usage of the Jacobian is not necessarily new (for example literature on adversarial samples or work on interpretability like GradCAM), casting these ideas through the lens of differential geometry was interesting. I found it interesting that Sim-Exp explores a far larger volumes compared to SiMEC. \n\n**Weaknesses**.\nWhile I found the mathematical tools that were developed to be sound, I found the experiments section were lacking many results and found it hard to justify the utility of the developed tools. The results on SimExp seem well understood from the lens of adversarial samples. There exist adversarial samples found around an epsilon-ball at every point so it isn't surprising that Sim-Exp can find such a sample. I also find results with SimEC somewhat underwhelming since one would expect that moving along the null-space of the Fisher would result in no changes to the outputs. With this context, I found objective (i) to be unsurprising.\n\nI found objective (ii) to be confusing. The results don't really suggest using the outputs of SimEXP or SimEC as alternative input data, but only that we can generate inputs / outputs from the same or different class. This does not suggest that we can use these inputs or outputs as additional training data. Could the authors clarify what they mean by objective (ii) and how the experiments are in support of this objective.\n\nIn general, I would have liked to seen an analysis comparing different architectures or more specific insights about the shape of the input manifolds and the equivalence classes. In my opinion, in its current state, the experiments don't really do justice to the tools developed in this work.", "was_revised": true, "final_justification": "I've increased my score after the rebuttal since some of the concerns regarding the framework were addressed. However, I think the paper lacks evidence showing that their theoretical framework and algorithms (SimEC, SimExp) can used to provide *new* insights about neural networks or the input data used to train them. \n\nI lean (only slightly) towards acceptance because the paper has interesting elements and there are exciting directions for future work. At the same time, I'm not extremely excited about the paper since the utility of their framework is mostly left to future work.", "rebuttal": "We thank the reviewer for their effort and thoughtful feedback on our work. We appreciate that the reviewer recognizes the novelty of casting our analysis through the lens of differential geometry. We would like to address their concerns and clarify the contributions of our experiments.\n\nThe central contribution of our paper, as the reviewer notes, is the employment of differential geometry. This allows our exploration algorithms to adapt directly to the learned manifold of the Transformer. Unlike methods that compute gradient-based updates in a presumed Euclidean space, our approach respects the specific, non-Euclidean geometry created by the model's series of transformations. This is achieved through the pullback metric, which precisely maps movement on the output manifold to the corresponding movement on the input manifold. This geometric awareness is also why a single exploration step in SiMExp can traverse a significantly larger volume than in SiMEC, even with an identical step multiplier ( $\\eta$ ), as it follows the natural curvature of the representation space.\nWe will now address the reviewer's specific questions.\n\n**Q1. Can the authors comment on the similarities or differences to literature on adversarial samples? Why is it surprising in this context that SiMExp works?**\n\nWhile both SiMExp and signed-gradient techniques leverage Jacobians, the substantial difference lies in our computation of the pullback metric. This component is fundamental, as it allows us to map an exploration step from the output manifold back to the input manifold, reflecting the true geometry of the model's learned space, even if not Euclidean.\n\nRegarding why the success of SiMExp is noteworthy: while we provide a solid mathematical foundation, empirical validation is essential. It is not a given that these theoretical tools will perform as expected on complex architectures like Transformers, especially considering numerical approximation challenges (e.g. float64 or float32). To demonstrate the practical utility of our modality-agnostic method, which is designed to work across Transformer architectures, we intentionally evaluated it on two distinct models, one for text and one for images. This cross-modal evaluation highlights the robustness and generality of our approach. Empirical evidence that our method performs well in practice is a necessary complement to its theoretical foundation. These results also provide insight into the model behaviors that our geometric tools are capable of revealing.\n\nOur primary objective in this work is to validate, both theoretically and empirically, the foundations of our geometric exploration framework applied to the Transformer architecture. Given this focus, we do not position our work as a direct competitor to methods aimed specifically at adversarial example generation or data augmentation. Instead, our methodology serves a different purpose: establishing a principled understanding of model behavior grounded in geometry. We see adversarial generation and augmentation as promising downstream applications of our approach, but not the focus of this initial study. In our view, building a solid and interpretable foundation is a necessary first step to enable stronger, more reliable applications in the future.\n\nThis leads directly to Objective (ii). The reviewer asked for clarification on this objective and its connection to the experiments. SiMEC and SiMExp are designed to generate new data points directly in the input embedding space, which can be used as alternative inputs in embedding form. However, this does not translate to generating corresponding samples such as new images or new texts.\nA natural next step is to project these alternative embeddings back into a human-interpretable space. This serves two key purposes. The first is qualitative interpretation: it allows us to visually inspect the generated data and observe how the exploration process alters the original input. This provides insight into what the model considers to belong to the \"same\" or \"different\" equivalence class, offering a glimpse into the structure of these classes (see token and patch changes in Figure 1). The second purpose is validation. This step is crucial to evaluate whether the human-interpretable counterparts can be used interchangeably with the embeddings from which they were generated, which involves analyzing the model's output when fed the decoded sample to understand its relationship with the output from the embedding itself.\n\n**Q2. Isn't it unsurprising that SiMEC is able to find very similar looking images that make the same predictions?**\n\nAs shown in literature (e.g., Shi et al., 2022, \"Decision-based black-box attack against vision transformers via patch-wise adversarial removal\", NIPS '22), even imperceptible changes can drastically alter a model's prediction if made in a specific, adversarial direction.\nIn this context, the contribution of SiMEC is its ability to perform a controlled exploration that selectively finds the input changes that maintain the output probability distribution. It successfully isolates directions of invariance from directions of high sensitivity. The fact that SiMEC automatically discovers the appropriate (and smaller) step size needed to stay within the same equivalence class—while SiMExp automatically finds a larger step to change it—is further empirical evidence validating our theoretical framework.\n\n**Q3. What is the distance traveled in weight space for SiMEC and SiMExp? Is the distance traveled by SiMEC extremely small, in which case aren't the results to be expected?**\n\nThis is an excellent question that points to a potential misunderstanding. Our methods, SiMEC and SiMExp, operate in the input embedding space, not the model's weight space. The model's weights remain frozen during our analysis. We apologize if this was not clear and will explicitly state it in the revised methods section.\n\nThe reviewer's question is likely about the distance traveled in the input space. As reported in lines 267-277, SiMExp explores a volume on average 10x larger than SiMEC. While the distance for SiMEC is smaller, the crucial point is the direction of travel, not just the magnitude. Adversarial literature shows that even infinitesimal steps can change the output if the direction is orthogonal to the decision boundary. SiMEC's contribution is finding the specific directions (tangent to the equivalence class manifold) where larger steps can be taken without changing the output. The fact that our geometrically-derived algorithms automatically discover these distinct scales of movement (small for SiMEC, large for SiMExp) further validates our framework.\n\nWe thank the reviewer again for their insightful feedback. We are confident that the revisions guided by their comments will significantly strengthen our paper's contribution by better framing it as a foundational work. Our goal is to validate a new theoretical framework for manifold exploration, which we see as a necessary first step for developing powerful future applications in areas like adversarial analysis and training augmentation."}, {"review_id": "de0MNZKe1C", "reviewer": "Reviewer_KszG", "summary": "The authors present an information geometric method for exploring how Transformer models encode their input data by identifying *equivalence classes*. i.e sets of inputs that yield the same or different predictive outcomes. In particular, the paper relies on Riemannian geometry to treat the layers of a Transformer as a sequence of transformations of the input manifold. The primary contributions are two algorithms:\n1. Singular Metric Equivalence Class (SiMEC): Identifying inputs that belong to the same equivalence class.\n2. Singular Metric Exploration (SiMExp): Complementary to SIMEC, finding perturbations to input that can change the model's prediction is important to understand the robustness of features. SIMEXP identifies inputs that to change equivalence classes, yielding a different probability distribution.\n\nAt a high level, the core insight is based on the spectral decomposition of a \"pullback metric\" that translates the geometry of the model's output space back to its input space. By moving along eigenvectors associated with zero eigenvalues (i.e. nullspace for SiMEC) or non-zero eigenvalues (for SiMExp), the algorithms can systematically explore the model's input manifold. The authors also demonstrates how these perturbed embeddings can be decoded back into human-readable formats like images and text, providing interpretable insights into the model's decision boundaries.", "questions": "Some questions that would help improving understanding the core contributions of the paper\n\n1. Computational overhead : Per my understanding, the eigendecomposition of the pullback metric has a computational complexity (per-token) of O(n^3). For modern architectures like ViTs, the models have embedding dimension in the order of 1024+. Could you please elaborate on potential ways to scale the method (e.g. via approximate eigendecompositions)? \n\n2. Impact of the feasible region projection: Could you share a more details on the design choices of the projection, such as the sensitivity of this step? E.g. how do the results change if a different projection method is used, such as projecting to the nearest neighbor in the training set's embedding space or onto a learned manifold (e.g., via a VAE)?\n\n3. generalizing the textual exploration beyond a single token : The experiments on the text modality provide a compelling proof-of-concept, but the scope is limited to modifying only the single token with the highest attribution value. Could the geometric information from the eigenvectors from this framework be leveraged to perform a more holistic exploration of textual inputs involving changes to multiple tokens?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "### **Strengths**\nSome strengths of the paper as submitted\n\n1.  **Principled and well-defined problem statement**: The paper's methodology builds on a robust and formally-defined foundation of Riemannian geometry. By treating a neural network as a sequence of maps between manifolds and using the pullback of the output space metric to define a singular Riemannian metric on the input space, the authors establish a formal framework to define equivalence classes based on zero pseudodistance ($Pd(x,y)=0$). \n\n2.  **Geometry based exploration across equivalence classes**: The **SiMEC** and **SiMExp** algorithms are a direct and innovative application of the paper's geometric theory. The insight of using the eigendecomposition of the pullback metric ($g^0_x$) and separating the exploration into directions within the kernel (eigenvectors of the zero eigenvalue) versus its complement (eigenvectors of non-zero eigenvalues) is useful for navigating within or across equivalence classes, respectively.\n\n3.  **Interpretability via input manifold exploration**: A significant contribution is the concrete method (Algorithm 3) for decoding explored embeddings back into a human-understandable format for both vision and text. The auhors provide practical approaches to recover such readouts based on certain constraints and assumptions.\n\n4.  **Empirical validation**: The paper's claims are validated across different data modalities (vision and text) and datasets (MNIST, CIFAR-10, WinoBias, and MHS). The experiments are well-designed to test the core hypotheses.\n\n\n### **Weaknesses**\n\n1.  **Computational overhead**: The primary limitation of the proposed method is its computational complexity. The reliance on eigendecomposition of the pullback metric has a complexity of $O(n^3)$, where *n* is the embedding dimension. This makes the approach computationally expensive and potentially impractical for state-of-the-art Transformers, which often have large embedding dimensions (e.g., 768, 1024, or higher).\n\n2.  **Potentially strong assumptions**: The practical implementation as described above involves several critical approximations that is likely to affect fidelity. The step size $\\delta^{(k)}$ is an estimate based on a local linearity assumption, and the projection onto a \"feasible region\" is a bounding box acknowledged as an over-approximation. These numerical errors can accumulate, leading to inconsistencies between the predictions from embeddings and the corresponding decodings might be misleading. \n\n3.  **Limitations to single-token exploration**: The text-based experiments are constrained to exploring modifications of only a **single token** with the highest attribution value. This is a significant simplification, as true semantic equivalence or difference in natural language often involves complex, combinatorial, or syntactic changes across an entire sentence.", "was_revised": true, "final_justification": "The paper's core contribution is its novel information-geometric framework for analyzing Transformer manifolds, which is technically sound. The rebuttal provides confidence that the main implementation limitations (scalability, fidelity) are addressable with future engineering, not inherent theoretical defects. I maintain my score at 4 (Borderline accept), reasons to accept outweigh reasons to reject.", "rebuttal": "We thank the reviewer for their insightful feedbacks on our work.  We appreciate that the reviewer acknowledges both our theoretical contributions and our empirical validation of the method we develope to explore how transformers encode their input data. The reviewer's remarks raised some interesting questions which we address below.\n\n**Q1. Computational overhead : Per my understanding, the eigendecomposition of the pullback metric has a computational complexity (per-token) of** $O(n^3)$ **. For modern architectures like ViTs, the models have embedding dimension in the order of** $1024+$ **. Could you please elaborate on potential ways to scale the method (e.g. via approximate eigendecompositions)?**\n\n In our work we decided to use the full decomposition of the metric to study our method in its entirety, thus considering all the information about the embedding space geometry carried by the spectral decomposition of the pullback metric. However, our algorithms can be modified to scale better with the embedding dimension. One possibility -- namely using an approximate eigendecomposition -- is suggested by the reviewer. While this method is indeed an option to make the problem scale better, it also introduces additional numerical errors at each step of the exploration process, whose accumulation may be not negligible in the long run. To avoid this problem, our method can be scaled making use of partial decompositions (such as Lacnzos algorithm and its variants like spectrum slicing techniques) computing e.g. only the first $k$ highest eigenvalues and associated eigenvectors (for SiMExp) or the lowest $k$ ones (for SiMEC). These methods, albeit potentially restricting the exploration of the embedding space to a smaller volume compared to the full decomposition of the metric, may also speed up the exploration process of in SiMEXP (the directions corresponding to the highest eigenvalues correspond to more modifications to the input). For SiMEC, limiting the eigendecomposition to the lowest eigenvalues can decrease the approximations error in the path built by SiMEC (the directions corresponding to the lowest eigenvalues correspond to less modifications of the input equivalence class) possibly at the expense of the speed of exploration.\n\n**Q2. Impact of the feasible region projection: Could you share a more details on the design choices of the projection, such as the sensitivity of this step? E.g. how do the results change if a different projection method is used, such as projecting to the nearest neighbor in the training set's embedding space or onto a learned manifold (e.g., via a VAE)?**\n\nWe agree with the reviewer's comment about our feasible region projection method that the projection to a bounding box is an over-approximation. However, it is a computationally inexpensive method which does not affect the speed of exploration and does not restrict the explored volume. Projecting to the nearest neighbor in the training set at each step reduces the speed and the volume of the exploration, since only significant enough changes of the input may be projected to different token embeddings. Small values of delta or proceeding in directions corresponding to small changes of the starting embedding may lead to a situation in which the projection leads back to the starting point, effectively preventing the exploration of the embedding space. Making use of a VAE to project to the feasible region is indeed an interesting suggestion which has been proposed for other (non-transformers) architectures employed to solve inverse problems in medical imaging. While worth experimenting with, in our work we decided to focus on the standard transformer architecture, hence the decision not to use additional networks to perform the projection.\n\nIn their comments, the reviewer also addressed another point related to this question, namely the validity of the hypotheses under which we perform the estimate of the step size $\\delta^{(k)}$. We think that the local linearity assumption is a reasonable hypothesis due to the fact that smooth manifolds are locally Euclidean and our metric is smooth (and therefore can be locally approximated by a Taylor expansion within a bounded error). While we agree with the reviewer that the estimate of the step size may introduce numerical errors, we note that this also happens e.g. in Euclidean ODEs integration algorithms such as the standard Euler method -- assuming that the solution can locally be approximated by a polygonal curve, which is indeed a local linearity assumption. We are aware that the accumulation of the numerical errors may lead to inconsistencies between the predictions from the embeddings and the corresponding decodings. We addressed this problem in Section 3.2 of our paper and in Section 6 of the Supplementary Materials, carrying out some numerical experiments to study the effect of decoding the embeddings produced by the algorithms.\n\n\n\n**Q3. Generalizing the textual exploration beyond a single token : The experiments on the text modality provide a compelling proof-of-concept, but the scope is limited to modifying only the single token with the highest attribution value. Could the geometric information from the eigenvectors from this framework be leveraged to perform a more holistic exploration of textual inputs involving changes to multiple tokens?**\n\nOur algorithms can indeed modify more than one token, as the eigenvector decomposition carries information on the whole sequence of tokens given in inputs. In our examples we decided to limit the exploration to one or two tokens for the sake of clarity, e.g. in Section 4, Figure 1 where only two tokens are modified. However, we considered modifications to all the tokens of a sentence in the analysis carried out in Section 6 of the Supplementary Materials."}, {"review_id": "jmFs7Hws6h", "reviewer": "Reviewer_JwKY", "summary": "The paper proposes a new interpretability tool for Transformers through the lens of Riemannian geometry. By modeling Transformer as smooth maps between manifolds, the authors compute the pullback of the output space metric onto the input space via the model’s Jacobian. This enables two complementary procedures: (1) identifying equivalence classes of inputs that yield the same output distribution, and (2) discovering directions that lead away from the current equivalence class. These resulting methods allow for principled exploration of the embedding space, enabling controlled and interpretable input perturbations.", "questions": "Those are written in the weakness section above.", "limitations": "Limitations are addressed throughout the paper.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\nThe use of differential geometric tools for Transformer analysis is a promising and underexplored direction in interpretability. This work contributes valuable theoretical insights to that space.\nThe proposed framework is modality-agnostic, demonstrated on both textual (BERT) and visual (ViT) architectures. Extending to other modalities such as audio or multimodal models would be a compelling future direction.\nThe projection of explored embeddings back into human-interpretable space (via decoding or inverse mapping) allows meaningful qualitative insight.\nThe mathematical formulation is rigorous and well presented, with clear derivations and thoughtful use of geometric formalism to support the validity of the method.\n\nWeaknesses\n-The methods works entirely in the embedding space and the projection back to input space is approximated and lossy. The methods practical impact e.g. for adversarial examples is therefore limited by the quality of the inverse mapping\n-The paper lacks comparisons to alternative interpretability methods. While the approach is conceptually distinct, a baseline or ablation comparison would better situate its effectiveness.\n-The authors state that Transformer components, such as fully connected layers, GeLU, and sigmoid activations, are smooth functions. Does that also hold true for other essential components, such as the softmax operation?\n-The method is evaluated only on one architecture per modality, and only on small or toy datasets. To support the claim of a general framework, broader empirical validation across architectures and tasks would be beneficial .\n\nMinor weakness: \nSome sentences are overly dense or overloaded with technical detail, affecting clarity (e.g. 140-143).  A clearer writing style would make the paper more accessible to a broader audience.", "was_revised": true, "final_justification": "I have read the rebuttal and the answers to my questions. I appreciate that the authors took the time to clarify my questions. I still believe it is above acceptance threshold, I keep my rating.", "rebuttal": "We thank the reviewer for the punctual remarks. As the precise questions are reported in the \"weaknesses\" section, we try to summarize and provide exhaustive answers to the points highlighted in there. \n\n**Q1. It is highlighted that projection from the embedding space to the input space is lossy**\n\nWe acknowledge this point in the paper, however we stress that one key point of our experiments is precisely to evaluate the difference produced by mapping the embedding vectors back into the input space through the interpretation step. Specifically, producing adversarial examples was not our primary objective, although it is an interesting application of our algorithms, just like sensitivity analysis and model explainability. From the empirical analysis we conducted in the paper, however, it is possible to support the claim that the theoretical framework can be applied effectively to real data, as well as to draw meaningful qualitative insights, such as visualization of changes in the input space (image patches or textual tokens in the experiments) by moving along relevant directions (i.e. those of maximum and null output change) in the embedding space.\nFurthermore, we believe human interpretability of the output of the algorithms to be an important feature as well, which can be achieved only through decoding, as it would be unfeasible to compute the pullback metric directly on high-dimensional input vectors such as vocabulary size for textual tokens. \n\n**Q2. Ablation study**\n\nAs for ablation study, since all steps of our algorithms are fundamental in its inner functioning, we refer the reviewer to tables from Section 6 onwards of the Supplementary Material, which provide a comparison between different choices of delta multiplier ($\\eta = 1$, which corresponds to eliminating the multiplier, and $\\eta > 1$, which aims at accelerating the exploration process, while we also experimented with $\\eta < 1$, without relevant results) and fraction of patches/tokens explored (a single patch/token, all patches/tokens simultaneously, half of them). \n\n**Q3. Is softmax a smooth function?**\n\nFinally, we remark that all components of most widespread Transformer Language and Vision Models are smooth functions (except for null-measure sets), including softmax and similar activations."}, {"review_id": "AalXVTRkr6", "reviewer": "Reviewer_AbP5", "summary": "The paper tackles the question “what does a Transformer consider the same?” by searching over the model’s input space using theory from Riemannian-geometry. Specifically, each layer is treated as a smooth deformation of a manifold, and the authors pull the output metric back to the input and slice that space into equivalence classes—groups of inputs the model assigns the same class-probability distribution. On top of this theory they build two algorithms: SiMEC, which move inside a class to surface to identify new, prediction-preserving examples, and SiMExp, which tries to move away from the current class to force the model toward a different label. They also show how to decode the explored embeddings into human-readable images or sentences, and run experiments on ViT and BERT variants with CIFAR-10, MNIST, WinoBias and hate-speech datasets to check speed, volume explored, and how often predictions flip.", "questions": "1. It is not entirely clear to me why we need to interpret the output of the algorithms. I understand why this is needed for textual data -- we optimize over continuous embedding but want discrete tokens. But for images, presumably the space is already continuous, so why not use the output as is?\n\n2. In text, are there examples in which multiple tokens change, or we see changes more meaningful than what is presented in Fig 1 (\"Mother\" --> \".\")?\n\n3. What do we learn from the WinoBias experiments? Can we shed some light on the way gender bias is manifested in the model?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 4, "strengths_and_weaknesses": "The paper studies an interesting question, aiming to induce minimal changes to a given input such as prediction either stays the same or flips. Beyond interpreting the input space and demonstrating how the model \"sees\" it -- i.e., which kind of changes correspond to output changes, and which do not -- being able to explore the model's (approximated) equivalence classes can be practically useful in debugging adversarial examples, e.g., showing that the model constructs its representation space in a non-robust way, where inputs which should be semantically equivalent end up in different equivalence classes. \n\nThe paper builds on mathematical theory, and the writing is quite dense. As someone with only rudimentary familiarity of the topic, it was quite difficult to understand the *meaning* of many of the mathematical formulations or their relevance. For instance, it was difficult to me to understand why the notion of pullback is relevant. I think I understand it now as a way to map the relation between inputs and outputs using the Jacobian, but this should be explained better. In terms of the experimental results, it is shown that the two algorithms are effective, in the sense that one of them roughly preserves output probabilities, while the others flips the class predictions. However, results are very limited, and beyond the average change in probability it is not clear whether the method is actually useful for any interpretability purpose. Do we can actually use it for debugging? What do we learn about the geometry of the input space? \n\nOne basic limitation of the proposed algorithms is that they optimize the continuous space, while we actually care about *semantic equivalence*. It's ok if the only goal is debugging -- the model might be convinced that slight pixel changes suddenly make an image of a cat look like an image of a dog -- but if the goal is to create human-interpretable adversarial examples, we would ideally be able to induce changes that bring us towards another equivalence class and go beyond slight noising of the original image.\n\nAnother issue with the evaluation is the lack of comparison with baselines. The two algorithms do achieve their goal, but one can think of other search methods that can conceivably also succeed, such as random gaussian noising and rejection sampling, gradient-guided search (without using second-order signal), etc. How do these compare with the proposed algorithms?", "was_revised": true, "final_justification": "The paper proposes a mathematically rigorous approach to understand how transformer models partition their input space. I initially had concerns about the writing and I did not understand some key points, which were clarified in the rebuttal. I hope the authors will revise the introduction according to their response.", "rebuttal": "We thank the reviewer for the precise remarks and questions. First of all, we want to shed more light on the general purpose of our work. The main objective of our work is to demonstrate an exploration method that adapts to the manifolds used by the transformer itself, with fewest assumptions as possible about their properties. In other words, our contribution is to theoretically formalize a new method for exploring how Transformers \"see\" the input space, respecting the metric of the output space. Furthermore, we want to demonstrate the empirical validity of the method using modern architecture and realistic tasks.\nThe second objective concerns the analysis of the behavior of our method in practical terms, which cannot be limited to the description of the embeddings obtained from the exploration, but must be extended to include a qualitative perception of what these embeddings represent in the input space understood by the model. As a consequence, the methods presented are not specifically tailored to interpretability of models, debugging, adversarial example generation, sensitivity analysis, or bias detection, but may be used for all these downstream tasks according to purpose, which is naturally an interesting future direction. As an example, in the field of adversarial example generation, our algorithms may be used to systematically build samples that are semantically equivalent for the model, which we claim to be as crucial as building samples that are semantically equivalent for the human point of view.\n\nWe now delve into the theoretical background. Other previous works exploit the Jacobian matrix of a Transformer model with the aim of exploring the input manifold. However, previous works focus exclusively on Euclidean spaces, whereas the relevance of the pullback metric relies precisely in the fact that it can be used to map the Jacobian, through equation (4) in the paper, into non-Euclidean spaces. As an example, we can think of hyperbolic-space Transformers (e.g. Ermolov et al, CVPR, 2022), in which the g matrix would correspond to a Lorentz matrix (Tabaghi \\& Dokmanic, SIGKDD, 2020). In the literature, few works have treated non-Euclidean output spaces in Transformers, respecting the true geometry of the model's mapping spaces.\n\nFinally, as for the experimental section, what we learn about the geometry of the input space is fundamentally in which directions, starting from an initial data point, we can move by keeping the output constant (i.e. remaining in the same equivalence class) and which are the directions that bring to a different equivalence class in the shortest ways. While difficult to visualize in human-understandable terms, due to the high dimensionality of the input space, this can be thought of as identifying what has to be changed (and how) to shift from one class to another (see for instance the token change and concentrated patch changes in Figure 1) and how may be changed without altering how the model sees the input (in Figure 1, for example, diffused changes in the image, or a word changed with a synonym as in Figure 3 in the Supplementary Material). \nThe lack of baselines is crucially due to the fact that other methods are not focused on remaining in the same equivalence class, while at most focusing on changing it assuming a Euclidean output space (gradient-guided search). While we did experiments with Gaussian noising, they revealed only that, though faster in computation, the method was useless in both objectives regarding equivalence class exploration, since it needed a huge number of iterations (at least in the order of 10000) to produce a dense enough exploration.\n\nPunctual answers to the questions are provided in the following:\n\n**Q1. It is not entirely clear to me why we need to interpret the output of the algorithms. I understand why this is needed for textual data -- we optimize over continuous embedding but want discrete tokens. But for images, presumably the space is already continuous, so why not use the output as is?**\n\nIn this case we are dealing with the output of the algorithms, i.e. the embeddings found in the exploration process, and not the output of the model, i.e. the probability distribution in the output space. While we also thought about the differences between continuous and discrete decoding of embeddings at the beginning of our work, we realized that the distinction between the two cases is unnecessary, as in both cases we can think of the final transformation as a decoding step. Indeed, a decoder is necessary in all cases, with any data type processed by the model. In the textual case, data are represented as one-hot encoded vectors, as such the decoder includes a final discretizing layer. In the visual case this step is not needed, however we still need a decoder, which in the most simple case corresponds to a pseudo-inverse matrix, that maps patch embeddings back to RGB tensors. Finally, we also consider that the image space may not be intended as continuous, but rather as a highly granular discrete space, since pixels cannot really assume continuous values. \n\n**Q2. In text, are there examples in which multiple tokens change, or we see changes more meaningful than what is presented in Fig 1 (\"Mother\" --> \".\")?**\n\nPremising that the examples, such as the one in Figure 1, report simple cases with one or two tokens modified at a time for the sake of clarity, we provide more complex experiments in the Supplementary Material, from section 5 onwards. In the tables therein presented, multiple token/patch configurations are analyzed. Specifically, for the MHS dataset: a single token, all tokens simultaneously, as well as half of them. The example provided in Figure 1 was most significant for SiMExp, while the reviewer may find an interesting example for SiMEC in Figure 3 of the Supplementary.\n\n**Q3. What do we learn from the WinoBias experiments? Can we shed some light on the way gender bias is manifested in the model?**\n\nAs previously stated, bias detection may be an interesting downstream task of our proposed approach. In here, WinoBias dataset is used as a practical, real-world example of the empirical validity of the SiMEC and SiMExp algorithms, not much for studying the problem of gender bias. In other words, while gender bias detection is certainly an interesting topic in terms of applicability of our approach, which we deem promising for the specific task, it would require further and more focused research."}]

Author rebuttal:

Meta-review: This paper proposes an approach for exploring the input space of Transformers using tools from differential geometry. The main idea is to model the network's layers as a series of deformations on the input manifold. By computing the pullback of the output space metric via the model's Jacobian, the authors define equivalence classes. They then propose the algorithms of SiMEC and SiMExp, which use the eigendecomposition of this pullback metric to navigate these equivalence classes, hence providing a principled way to interpret the model's perception.

In terms of strengths, reviewers noted the paper's novel and principled theoretical framework. The aproach was highlighted as a promising, sound, and innovative  (bq1s, KszG, JwKY, AbP5). The mathematical formulation was considered rigorous and well-presented (bq1s, JwKY). The framework was noted for its practical advantages, including being modality-agnostic and providing meaningful, interpretable outputs by decoding explored embeddings (KszG, JwKY).

Inititially, a shared concerns was the lack of demonstrated utility and missing baslines. Reviewers argued the experiments were limited to validating the method's mechanics and did not show what new insights could be learned about the model or how the generated inputs could be used meaningfully for downstream tasks (bq1s, AbP5). The absence of a simple baseline, such as a random Gaussian walk,  was also noted (bq1s, AbP5). Other concerns included the computational overhead limited scope of the experiments (KszG, JwKY).

The post-rebuttal discussion solidified a clear but conflicted consensus among the reviewers. On one hand, reviewers acknowledged that the paper presents a "technically sound" and "novel information-geometric framework" (KszG) with "interesting elements and... exciting directions for future work" (bq1s). On the other hand, they were unenthusiastic, noting that the "utility of their framework is mostly left to future work" and that it "lacks evidence showing that [it] can used to provide new insights" (bq1s). The AC believes that while the immediate utility is not shown, the high-quality foundational methodology is novel, sound, and promising. Therefore, the AC finds that reasons to accept outweigh reasons to reject and recommends acceptance.
Decision: Accept (poster)