Title: Unveiling Transformer Perception by Exploring Input Manifolds
Venue: NeurIPS 2024
Accepted: no
Keywords: transformers interpretability

input space exploration

geometric deep learning
Abstract: This paper introduces a general method for the exploration of equivalence classes in the input space of
Transformer models. The proposed approach is based on sound mathematical theory which describes the internal layers of a Transformer architecture as sequential deformations of the input manifold. Using eigendecomposition of the pullback of the distance metric defined on the output space through the Jacobian of the model, we are able to reconstruct equivalence classes in the input space and navigate across them. We illustrate how this method can be used as a powerful tool for investigating how a Transformer sees the input space, facilitating local and task-agnostic explainability in Computer Vision and Natural Language Processing tasks.

1

Abstract

This paper introduces a general method for the exploration of equivalence classes in the input space of Transformer models. The proposed approach is based on sound mathematical theory which describes the internal layers of a Transformer architecture as sequential deformations of the input manifold. Using eigendecomposition of the pullback of the distance metric defined on the output space through the Jacobian of the model, we are able to reconstruct equivalence classes in the input space and navigate across them. We illustrate how this method can be used as a powerful tool for investigating how a Transformer sees the input space, facilitating local and task-agnostic explainability in Computer Vision and Natural Language Processing tasks.

1 Introduction

2 Preliminaries 32

33

34

In this Section, we provide the theoretical foundation of the proposed approach, namely the Geometric Deep Learning framework based on Riemannian Geometry [2].

35

36

37

38

39

40

41

42

43

44

45

46

47

48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

A neural network is considered as a sequence of maps, the layers of the network, between manifolds, and the latter are the spaces where the input and the outputs of the layers belong to.

Definition 1 (Neural Network) . A neural network is a sequence of C 1 maps Λ i between manifolds of the form:

We call M 0 the input manifold and M n the output manifold. All the other manifolds of the sequence are called representation manifolds. The maps Λ i are the layers of the neural network. We denote with N ( i ) = Λ n ◦ · · · ◦ Λ i : M i → M n the mapping from the i -th representation layer to the output layer.

As an example, consider a shallow network with just one layer, the composition of a linear operator A · + b with a sigmoid function σ , where A ∈ R m × n and b ∈ R m : then, the input manifold M 0 and the output manifold M 1 shall be R n and R m , respectively, and the map Λ 1 ( · ) = σ ( A · + b ) . We generalize this observation into the following definition.

Definition 2 (Smooth layer) . A map Λ i : M i -1 → M i is called a smooth layer if it is the restriction to M i -1 of a function Λ ( i ) ( x ) : R d i -1 → R d i of the form

for i = 1 , · · · , n , x ∈ R d i , b ( i ) ∈ R d i and A ( i ) ∈ R d i × d i -1 , with F ( i ) : R d i → R d i a diffeomorphism.

Remark 1. Transformers implicitly apply for this framework, since their modules are smooth functions, such as fully connected layers, GeLU and sigmoid activations.

Our aim is to transport the geometric information on the data lying in the output manifold to the input manifold: this allows us to obtain insight on how the network "sees" the input space, how it manipulates it for reaching its final conclusion. For fulfilling this objective, we need several tools from differential geometry. The first key ingredient is the notion of singular Riemannian metric, which has the intuitive meaning of a degenerate scalar product which changes point to point.

Definition 3 (Singular Riemannian metric) . Let M = R n or an open subset of R n . A singular Riemannian metric g over M is a map g : M → Bil ( R n × R n ) that associates to each point p a positive semidefinite symmetric bilinear form g p : R n × R n → R in a smooth way.

Without loss of generality, we can assume the following hypotheses on the sequence (1): i) The manifolds M i are open and path-connected sets of dimension dim M i = d i . ii) The maps Λ i are C 1 submersions. iii) Λ i ( M i -1 ) = M i for every i = 1 , · · · , n . iv) The manifold M n is equipped with the structure of Riemannian manifold, with metric g ( n ) . Definition 3 naturally leads to the definition of the pseudolength and of energy of a curve.

Definition 4 (Pseudolength and energy of a curve) . Let γ : [ a, b ] → R n a curve defined on the interval [ a, b ] ⊂ R and ∥ v ∥ p = √ g p ( v, v ) the pseudo-norm induced by the pseudo-metric g p at point p . Then the pseudolength of γ and its energy are defined as

The notion of pseudolength leads naturally to define the distance between two points. 69

Definition 5 (Pseudodistance) . Let x, y ∈ M = R n . The pseudodistance between x and y is then 70

One can observe that endowing the space R with a singular Riemannian metric leads to have 71

non trivial curves whose length is zero. A straightforward consequence is that there are distinct 72

points whose pseudodistance is therefore zero: a natural equivalence relation arises, i.e. x ∼ y ⇔ 73

75

76

n n

The second crucial tool is the notion of pullback of a function. Let and fix the coordinate systems

x

= (

x

, . . . , x

)

and

y

= (

y

, . . . , y

f

)

be a function from on

R

p

and on

R

q

R

p

to

R

q

,

, respectively.

Moreover, we endow R q with the standard Euclidean metric g , whose associated matrix is the identity. 77

1

p

1

q

The space R p can be equipped with the pullback metric f ∗ g whose representation matrix reads as 78

The sequence (1) shows that a neural network can be considered simply as a function, a composition 79 of maps: hence, taking f = Λ n ◦ Λ n -1 ◦ · · · ◦ Λ 1 and supposing that M 0 = R p , M n = R q , the 80 generalization of (5) applied to (1) provides with the pullback of a generic neural network. 81

q ( n )

Hereafter, we consider in (1) the case M n = R , equipped with the trivial metric g = I q , i.e. , 82 the identity. Each manifold M i of the sequence (1) is equipped with a Riemannian singular metric, 83 denoted with g ( i ) , obtained via the pullback of N ( i ) . The pseudolength of a curve γ on the i -th 84 manifold, namely Pl i ( γ ) , is computed via the relative metric g ( i ) via (3). 85

86

87

88

89

90

91

92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

2.1 General results

Wedepict hereafter the theoretical bases of our approach. We denote with N i the submap Λ i ◦· · ·◦ Λ n : M i → M n , and with N ≡ N 0 the map describing the action of the complete network. The starting point is to consider the pair ( M i , P d i ) : this is a pseudometric space, which can be turned into a full-fledged metric space M i / ∼ i by the metric identification x ∼ i y ⇔ Pd i ( x, y ) = 0 . The first result states that the length of a curve on the i -th manifold is preserved among the mapping on the subsequent manifolds.

Proposition 1. Let γ : [0 , 1] → M i be a piecewise C 1 curve. Let k ∈ { i, i +1 , · · · , n } and consider the curve γ k = Λ k ◦ · · · ◦ Λ i ◦ γ on M k . Then Pl i ( γ ) = Pl k ( γ k ) .

In particular this is true when k = n , i.e. , the length of a curve is preserved in the last manifold. This result leads naturally to claim that if two points are in the same class of equivalence, then they are mapped into the same point under the action of the neural network.

Proposition 2. If two points p, q ∈ M i are in the same class of equivalence, then N i ( p ) = N i ( q ) .

The next step is to prove that the sets M i / ∼ i are actually smooth manifolds: to this aim, we introduce another equivalence relation: x ∼ N i y if and only if there exists a piecewise γ : [0 , 1] → M i such that γ (0) = x, γ (1) = y and N i ◦ γ ( s ) = N i ( x ) ∀ s ∈ [0 , 1] . The introduction of this equivalence relation allows us to easily state the following proposition.

Proposition 3. Let x, y ∈ M i , then x ∼ i y if and only if x ∼ N i y .

The following corollary contains the natural consequences of the previous result; the second point of the claim below is the counterpart of Proposition 2.

Corollary 1. Under the hypothesis of Proposition 3, one has that M i / ∼ i = M i / ∼ N i +1 . Moreover, if two points p, q ∈ M i are connected by a C 1 curve γ : [0 , 1] → M i satisfying N i ( p ) = N i ◦ γ ( s ) for every s ∈ [0 , 1] , then they lie in the same class of equivalence.

Making use of the Godement's criterion, we are now able to prove that the set M i / ∼ i is a smooth manifold, together with its dimension.

Proposition 4. M i ∼ i is a smooth manifold of dimension dim ( N ( M 0 )) .

This last achievement provides practical insights about the projection π i on the quotient space, that consists the building block of the algorithms used for recovering and exploring the equivalence classes of a neural network.

Proposition 5. π i : M i → M i / ∼ i is a smooth fiber bundle, with Ker ( dπ i ) = V M i , which is 115 therefore an integrable distribution. V M i is the vertical bundle of M i . Every class of equivalence 116 [ p ] is a path-connected submanifold of M i and coincide with the fiber of the bundle over the point 117 p ∈ M i . 118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

3 Methodology

The results depicted in Section 2.1 provide powerful tools for investigating how a neural network sees the input space starting from a point x . In particular we point out the following remarks: i) If two points x, y belonging to the input manifold M 0 are are such that x ∼ 0 y , then N ( x ) = N ( y ) ; ii) given a point p ∈ M n , the counterimage N -1 ( p ) is a smooth manifold, whose connected components are classes of equivalences in M 0 with respect to ∼ 0 . Anecessary condition for two points x, y ∈ M 0 to be in the same class of equivalence is that N ( x ) = N ( y ) ; iii) any class of equivalence [ x ] , x ∈ M 0 , is a maximal integral submanifold of V M 0 . The above observations directly provide with a strategy to build up the equivalence class of an input point x ∈ M 0 . Proposition 5 tells us that V M 0 is an integrable distribution, with dimension equal to the dimension of the kernel of g (0) : we can hence find dim ( Ker ( g (0) )) vector fields which are a base for the tangent space of M 0 . This means that we can compute the eigenvalue decomposition of g (0) x and consider the L linearly independent eigenvectors, namely { v l } l =1 ,...,L , associated to the null eigenvalue: these eigenvectors depend smoothly on the point, a fact that is not trivial when the matrix associated to the metric depends on several parameters [15]. We can build then all the null curves by randomly selecting one eigenvector ˜ v ∈ { v l } and then reconstruct the curve along the direction ˜ v from the starting point x . From a practical point of view, one is led to solve the Cauchy problem, a first order differential equation, with ˙ γ = ˜ v and initial condition γ (0) = x .

3.1 Input Space Exploration

This whole procedure is coded in the Singular Metric Equivalence Class (SiMEC) and the Singular Metric Exploration (SiMExp) algorithms, whose general schemes are depicted in Algorithms 1 and 2. SiMEC reconstructs the class of equivalence of the input via the exploration of the input space by randomly selecting one of the eigenvectors related to the zero eigenvalue. On the opposite, in SiMExp, in order to move from a class of equivalence to another we consider the eigenvectors relative to the nonzero eigenvalues. This requires the slight difference in lines 5 to 7 between Algorithm 1 and Algorithm 2.

Algorithm 1 The Singular Metric Equivalence Class (SiMEC) algorithm.

1: Set the network N ; choose the maximum number of iterations. Choose the input p 0 .

2: for k = 0 , 1 , . . . , K -1 do

3: Compute g n N ( p k )

5: Diagonalize g 0 p k and find the eigenvectors { v l } l associated to the zero eigenvalue

4: Compute the pullback metric g 0 p k

6: Randomly select ˜ v ∈ { v l } l

7: δ = 1 / √ max( eigenvalues of g 0 p k )

8:

p

k

+1

p

k

+

δ

˜

v

9: end for

10: Optionally: store { p k } k =0 ,...,K for optimizing future computations

11: Project p k to the nearest feasible region

Algorithm 2 The Singular Metric Exploration (SiMExp) algorithm.

1: Set the network N ; choose the maximum number of iterations. Choose the input p 0 .

2: for k = 0 , 1 , . . . , K -1 do

3: Compute g n N ( p k )

5: Diagonalize g 0 p k and find the eigenvectors { w l } l associated to the non-zero eigenvalue

4: Compute the pullback metric g 0 p k

6: Randomly select ˜ w ∈ { w l } l

7: δ = 2 / √ max( eigenvalues of g 0 p k )

8: p k +1 ← p k + δ ˜ w

9: end for

10: Optionally: store { p k } k =0 ,...,K for optimizing future computations

11: Project p k to the nearest feasible region

146 pullback may lead to have even negative eigenvalues: hence one may use the notion of energy of 147 a curve, related to the pseudolength. The update rule for the new point (line 8) amounts to solve 148 the differential problem via the Euler method: for a reliable solution, we suggest to choose a small 149 step-length δ . On the other hand, if the value of δ is too small more iterations are needed to move 150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

away from the starting point sensibly. Therefore there is a trade-off between the reliability of the solution and the exploration pace. The proof of the well-posedness theorem for Cauchy problems, cf. [18, Theorem 2.1], yields some insights, suggesting to set δ equal to the inverse of the Lipschitz constant of the map N - which in practice we can estimate with the inverse of the square root of the largest eigenvalue λ M of the pullback metric g 0 p k . This is our default choice for Algorithm 1. We also note that Algorithm 1 is more sensitive to the choice of the parameter δ compared to Algorithm 2. To build points in the same equivalence class Algorithm 1 needs to follow a null curve closely with as little approximations as possible, namely with a small δ . In contrast Algorithm 2, whose goal is to change the equivalence class from one iteration to the next, does not have the same problem and larger δ are allowed. Out default choice is therefore to set δ = 2 λ -1 / 2 M for Algorithm 2. As for the computational complexity of the two algorithms, the most demanding step is the computation of the eigenvalues and eigenvectors, which is O ( n 3 ) , with n the dimension of the square matrix g 0 p k [20]. Since all the other operations are either O ( n ) or O ( n 2 ) , we conclude that the complexity of both Algorithms 1 and 2 is O ( n 3 ) .

3.2 Interpretability

Algorithms 1 and 2 allow for the exploration of the equivalence classes in the input space of a Transformer model. However, the points explored by these algorithms may not be directly interpretable by a human perspective. For instance, an image or a piece of text may need to be decoded to be 'readable' by a human observer. Furthermore, we present an interpretation of the eigenvalues of the pullback metric which allows us to define a feature importance metric. We present two interpretability methods for Transformers based on input space exploration. Both methods are then demonstrated on a Vision Transformer (ViT) trained for digit classification [8], and two BERT models, one trained for hate speech classification and the other trained for MLM [7, 19].

Algorithm 3 Feature Importance Analysis Using Pullback Metric g 0 x e

1: Inputs:

2: Transformer model T with: Tokenizer t T , Embedding layer e T , Intermediate layers l T 3: Input data x

4: Tokenize input x to obtain tokens x t = t T ( x )

6: Compute intermediate representations g n l T ( x e )

5: Compute embeddings x e = e T ( x t )

7: Calculate the pullback metric g 0 x e

8: Diagonalize g 0 x e to extract eigenvalues

9: Identify the maximum eigenvalue for each embedding, indicating its importance

10: Output: Heatmap of embedding importance based on the eigenvalues

Algorithm 4 Exploration of Embedding Space in Transformers

1: Inputs:

2: Transformer model T with: Tokenizer t T , Embedding layer e T , Intermediate layers l T

3: Input data x (image or text)

4: Retrieve segments x t = t T ( x ) .

5: Choose segments P = { p | p ∈ x t } for updates; keep others unchanged.

6: Compute embeddings x e = e T ( x t ) .

7: Apply SiMEC or SiMExp on x e , updating embeddings for segments in P .

8: Outputs: Modified input embedding, one for each SiMEC/SiMExp iteration.

Feature importance. Consider a Transformer model T whose architecture includes a tokenizer t T 175 (or patcher for images) that segments the input so that each segment can be converted into a continuous 176 representation by an embedding layer e T . This results in a matrix of dimensions n s × h , where n s 177 represents the number of segments, and h denotes the hidden size of the model's embeddings. The 178 eigenvalues of the pullback metric can be used to deduce the importance of each embedding and, by 179 extension, the significance of the segments they represent, with respect to the final prediction. The 180 process for determining the importance of textual tokens or image patches is outlined in Algorithm 3. 181 The appearance of the resulting heatmaps varies according to the type of input used. An example 182 of experiments with ViT on the MNIST dataset [12] is shown in Figure 1 that depicts heatmaps for 183 two MNIST instances. Figure 2, on the left, illustrates two experiment using Algorithm 3 on both a 184 BERT model for hate speech detection and a BERT model for MLM. 185

186

187

188

189

Interpretation of input space exploration. Using SiMEC and SiMExp to explore the embedding space reveals how Transformer models perceive equivalence among different data points. Specifically, these methodologies facilitate the sequential acquisition of embedding matrices p 0 . . . p K at each iteration, as detailed in Algorithms 1 and 2. Algorithm 4 implements a practical application of the

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

120

50

Eigenvalues

中

982

Eigenvalues

20

10

[CLS]ilikepizzawith

[MASK]

and

cheese

[SEP]

[CLS]ilikepizzawith

[MASK]

and

cheese

[SEP]

1

5

8

1

2

3

8

ImportanceLevel

ImportanceLevel

SiMEC/SiMExp approach with Transformer models. A key feature of this method is its ability to selectively update specific tokens (for text inputs) or patches (for image inputs) during each iteration. This selective updating allows us to explore targeted modifications that prompt the model to either categorize different inputs as the same class or recognize them as distinct. Unlike traditional approaches where modifications are predetermined, this method lets the model itself guide us to understand which data points belong to specific equivalence classes. To interpret embeddings resulted from the exploration process, they must be mapped back into a human-understandable form, such as text or images. The interpretation of an embedding vector depends on the operations performed by the Transformer's embedding module e T . If e T consists only of invertible operations, it is feasible to construct a layer that performs the inverse operation relative to e T . The output can then be visualized and directly interpreted by humans, allowing for a comparison with the original input to discern how differences in embeddings reflect differences in their representations (e.g., text, images). If the operations in e T are non-invertible, a trained decoder is required to reconstruct an interpretable output from each embedding matrix p 0 . . . p K . When using a BERT model, it is feasible to utilize layers that are specialized for the masked language modeling (MLM) task to map input embeddings back to tokens. This approach is effective whether the BERT model in question is specifically designed for MLMor for sentence classification. In the case of sentence classification models, it is necessary to select a corresponding MLM BERT model that shares the same internal architecture, including the number of layers and embedding size.

Algorithm 5 depicts the process of interpreting Algorithm 4 outputs for both ViT and BERT experiments. After initializing the decoder according to the model type, the embeddings p 0 . . . p K need to be constrained to a feasible region. This region is defined by the distribution of embeddings derived from the original input instances. Next, the embeddings are decoded, and the selected segments for exploration are extracted. These segments are then used to replace the corresponding parts of the original input instance. Figure 3 depicts an example outcome of Algorithm 5 applied on a ViT exploration experiment. Given that the interpretation process includes both a capping step and a decoding step (lines 10 and 11 of Algorithm 5), it's important to note that there isn't a direct 1:1 correspondence between each iteration's update and the interpretation outcomes. Our primary focus is on exploring the input embedding space, rather than the input image or input sentence spaces. For further investigation, we provide a detailed discussion on considering interpretation outputs as alternative prompts in Section 4.

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

Algorithm 5 Interpretation for Exploration results for ViT and BERT models.

1: Inputs:

2: Transformer model T with: Tokenizer t T , Embedding layer e T , Intermediate layers l T

3: Modified embeddings p 0 . . . p K resulted from Algorithm 4 applied on an input x

4: P = { p | p ∈ x t } indices of updated segments

5: If T is ViT:

6: Initialize decoder d with weights from e T .

7: If T is BERT:

8: Initialize decoder with intermediate and final layers of a BERT for MLM task.

9: Compute embeddings distributions for original input data

10: Use the original embeddings distributions to cap p 0 . . . p K

12: For each x ′ ∈ X ′ : replace segments relative to indices P in x with those in x ′ .

11: Decode modified embeddings p 0 . . . p K using d to generate the corresponding images/sentences X ′ = x ′ 0 . . . x ′ K .

13: Outputs:

14: Modified input images/sentences, one for each SiMEC/SiMExp iteration.

4 Experiments

Experiments are conducted on textual and visual data. We aim to perform a preliminary investigation of 3 features of our approach: (i) how the class probability changes on the decoded output of SiMEC/SiMExp, (ii) what is the trade-off between the quantity and the quality of the output, and (iii) how our method can be used to extract feature importance-based explanations.

In the textual case, we experiment with hate speech classification datasets: we use HateXplain 2 [13], which provides a ground truth for feature importance, plus a sample of 100 hate speech sentences generated by prompting ChatGPT 3 , which serve purposes (i) and (ii). In the visual case, we perform experiments on MNIST [12] dataset.

Using interpretation outputs as alternative prompts An interesting investigation is to determine if our interpretation algorithm (Algorithm 5) can generate alternative prompts that stay in the same equivalence class as the original input data or move to a different one, based on SiMEC and SiMExp explorations. We test how the probability assigned to the original equivalence class by the Transformer model changes as the SiMEC and SiMExp algorithms explore the input embedding manifold.

̸

For BERT experiments we generate prompts to inspect the probability distribution over the vocabulary for tokens updated by Algorithms 1 and 2. We decode the updated p 0 . . . p K using Algorithm 5, focusing on tokens updated through the iterations. For each of these decoded tokens, we extract the top-5 scores to obtain 5 alternative tokens to replace the original ones, creating 5 alternate prompts. We then extract the prediction i ∗ = arg max i y i for the original sentence, which represents the output whose equivalence class we aim to explore. Finally, we classify the new prompts, obtaining the corresponding predictions Y = y (0) . . . y ( K ) , where each y ( k ) ∈ R N , N being the number of prediction classes. We visualize the prediction trend for the i ∗ th value in every y (0) . . . y ( K ) categorizing the images into two subsets: those that lead to a change in prediction Y c = { y ( k ) ∈ Y | arg max i y ( k ) i = i ∗ } and those that don't Y s = { y i ∈ Y | arg max i y ( k ) i = i ∗ } .

EquivalenceclassprobabilityvalueforyEYc

EquivalenceclassprobabilityvalueforyEYs

BERTforHateSpeechDetection

BERTforHateSpeechDetection

0.35

algorithm

simec

0.94

algorithm

simec

0.30

simexp

0.92

simexp

0.25

Probability

Probability

0.90

0.20

0.88

0.15

0.10

0.86

0.05

0.84

0.00

0

200

400

600

800

1000

0

200

400

600

800

1000

Iteration

Iteration

Sentence classification experiments 4 involved 1000 iterations from both SiMEC and SiMExp, applied 245 to a subset of 8 sentences from the ChatGPT hate speech dataset. The plot on the left side of Figure 4 246 illustrates that, as the original embeddings are increasingly modified, SiMExp tends to produce 247 alternatives with lower prediction values for i ∗ compared to SiMEC. Thus, even if predictions change 248 in SiMEC experiments, the equivalence class prediction value remains approximately constant and 249 higher than in SiMExp. Considering the plot on the right side of Figure 4, SiMExp identifies prompts 250 that lower the prediction value for i ∗ . ViT and MLM experiments are detailed in the Supplementary 251 Materials. 252

253

254

255

256

257

258

259

260

261

Input space exploration We measure the time required to explore the input space of a ViT with the SiMEC algorithm and compare it with a perturbation-based method. The perturbation-based method mimics a trial-and-error approach as it takes an input image and, at each iteration, perturbs it by a semi-random vector v t +1 = a t v t + ηϵ , where a t = 1 if y t = y t -1 , a t = -1 otherwise, ϵ is an orthogonal random vector from a standard normal distribution and η is the step length. With the perturbation, we obtain a new image, then check whether the model yields the same label for the new image. The perturbation vector is re-initialized at random from a normal distribution 20% of the times to allow for exploration. We construct this method to have a direct comparison with ours in the absence of a consolidated literature about the task.

262

263

264

265

266

267

268

269

270

271

272

273

274

275

276

277

278

279

We train a ViT model having 4 layers and 4 heads per layer on the MNIST dataset 5 . The SiMEC algorithm is run for 1000 iterations, so that it can generate 1000 examples starting from a single image. In a sample of 100 images, the average time is approximately 339 seconds. 6 In the same time, the perturbation-based algorithm can produce up to 36000 images. However, we notice that the perturbation-based algorithm ends up producing monochrome (pixel color has zero variance) or totally noisy images, which provide little information about the behavior of the model. Excluding only the images with low color variance ( < 0 . 01 ), we are left, on average, with 19 images (standard deviation 13 . 9 ). SiMEC, in contrast, doesn't present this behavior, as all 1000 images have high enough intensity variance and are thus useful for explainability purposes.

As BERT has many more parameters with respect to our ViT model, processing textual data takes longer. Specifically, in a sample of 16 sentences, the average time needed to run 1000 iterations on a sentence is 7089 seconds, taking into account both MLM and classification experiments.

Feature importance-based explanations We compare our method against Attention Rollout (AR) [1] and the Relevancy method proposed by Chefer et al. [6]. In the textual case, we provide a quantitative evaluation using the HateXplain dataset, which contains 20147 sentences (of which 1924 in the test set) annotated with normal , offensive and hate speech labels as well as the positions of words that support the label decision. We then measure the cosine similarity between the importance assigned by each method to each word in a sentence and the ground truth. Notice that, since the

280

281

282

283

284

285

286

287

288

289

290

291

292

293

294

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

dataset contains multiple annotations, the ground truth y for each word w is obtained as the average of the binary labels assigned by each annotator, and therefore y ( w ) ∈ [0; 1] . We also normalize all scores in [0; 1] so to have them on the same scale. The average similarity achieved by our method is 0.707 (standard deviation σ = 0.302), against 0.7 ( σ = 0.315) for Relevancy and 0.583 ( σ = 0.318) for AR. This proves our method to be more effective in finding the most sensitive tokens for classification. We provide an example on image classification in the Supplementary Materials.

5 Related work

Our work relates to embedding space exploration literature, and has at least one collateral applications in the XAI domain, namely producing feature importance-based explanations.

Embedding space exploration. Works dealing with embedding space exploration mostly focus on the study of specific properties of the embedding space of Transformers, especially in NLP. For instance, Cai et al. [5] challenge the idea that the embedding space is inherently anisotropic [10] discovering local isotropy, and find low-dimensional manifold structures in the embedding space of GPT and BERT. Bi´ s et al. [3] argue that the anisotropy of the embedding space derives from embeddings shifting in common directions during training. In the field of CV, Vilas et al. [21] map internal representations of a ViT onto the output class manifold, enabling the early identification of class-related patches and the computation of saliency maps on the input image for each layer and head. Applying Singular Value Decomposition to the Jacobian matrix of a ViT, Salman et al. [17] treat the input space as the union of two subspaces: one in which image embedding doesn't change, and another one for which it changes. Except for the last one, all the aforementioned approaches rely on data samples. By studying the inverse image of the model, instead, we can do away with data samples.

Feature importance-based explanations. Feature importance is a measure of the contribution of each data feature to a model prediction. In the context of Computer Vision and Natural Language Processing, it amounts to giving a weight to pixels (or patches of pixels) in an image and tokens in a piece of text, respectively. In recent years, much research has focused on Transformers in both CV and NLP. Most approaches are based on the attention mechanism of the Transformer architecture. Abnar and Zuidema [1] quantify the overall attention of the output on the input by computing a linear combination of layer attentions (Attention Rollout) or applying a maximum flow algorithm (Attention Flow). To overcome the limitations [4] of attention-based methods, Hao et al. [11] use the concept of attribution , which is obtained by multiplying attention matrices by the integrated gradient of the model with respect to them. Chefer et al. [6] propose the Relevancy metric to generalize attribution to bi-modal and encoder-decoder architectures. Other methods are perturbation-based, where perturbations of input data are used to record any change in the output and draw a saliency map on the input. In order to overcome the main issue with such methods, i.e. the generation of outlier inputs, Englebert et al. [9] apply perturbations after the position encoding of the patches. In contrast with these methods, ours does not need arbitrary perturbations of inputs, and considers all parameters of the model, not only the attention query and key matrices.

6 Conclusions

Our exploration of the Transformer architecture through a theoretical framework grounded in Riemannian Geometry led to the application of our two algorithms, SiMEC and SiMExp, for examining equivalence classes in the Transformers' input space. We demonstrated how the results of these exploration methods can be interpreted in a human-readable form and conducted preliminary investigations into their potential applications. Notably, our methods show promise for ranking feature importance and generating alternative prompts within the same or different equivalence classes.

Future research directions include expanding our experimental results and delving deeper into the potential of our framework for controlled input generation within an equivalence class. This application holds significant promise for enhancing the explainability of Transformer models' decisions and for addressing issues related to bias and hallucinations.

329

330

331

332

333

334

335

336

337

338

339

340

341

342

343

344

345

346

347

348

349

350

351

352

353

354

355

356

357

358

359

360

361

362

363

364

365

366

367

368

369

370

371

372

373

374

375

References

S. Abnar and W. Zuidema. Quantifying attention flow in transformers. 2020. arXiv:2005.00928.

A. Benfenati and A. Marta. A singular Riemannian geometry approach to Deep Neural Networks I. Theoretical foundations. Neural Networks , 158:331-343, 2023.

D. Bi´ s, M. Podkorytov, and X. Liu. Too much in common: Shifting of embeddings in transformer language models and its implications. In Proceedings of the 2021 conference of the North American chapter of the Association for Computational Linguistics: Human Language Technologies , pages 5117-5130, 2021.

G. Brunner, Y. Liu, D. Pascual, O. Richter, M. Ciaramita, and R. Wattenhofer. On identifiability in transformers. International Conference on Learning Representations , 2019.

X. Cai, J. Huang, Y. Bian, and K. Church. Isotropy in the contextual embedding space: Clusters and manifolds. In International conference on learning representations , 2020.

H. Chefer, S. Gur, and L. Wolf. Generic attention-model explainability for interpreting bi-modal and encoder-decoder transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 397-406, 2021.

J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In J. Burstein, C. Doran, and T. Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) , pages 4171-4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19-1423. URL https://aclanthology.org/N19-1423 .

A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale, 2021. arXiv:2010.11929.

A. Englebert, S. Stassin, G. Nanfack, S. A. Mahmoudi, X. Siebert, O. Cornu, and C. De Vleeschouwer. Explaining through transformer input sampling. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 806-815, 2023.

J. Gao, D. He, X. Tan, T. Qin, L. Wang, and T.-Y. Liu. Representation degeneration problem in training natural language generation models. In International conference on learning representations , volume abs/1907.12009, 2019. URL https://api.semanticscholar.org/ CorpusID:59317065 .

Y. Hao, L. Dong, F. Wei, and K. Xu. Self-attention attribution: Interpreting information interactions inside transformer. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 35, pages 12963-12971, 2021.

Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE , 86(11):2278-2324, 1998. doi: 10.1109/5.726791.

B. Mathew, P. Saha, S. M. Yimam, C. Biemann, P. Goyal, and A. Mukherjee. Hatexplain: A benchmark dataset for explainable hate speech detection. In Proceedings of the AAAI conference on artificial intelligence , volume 35, pages 14867-14875, 2021.

N. Papernot, P. McDaniel, S. Jha, M. Fredrikson, Z. B. Celik, and A. Swami. The Limitations of Deep Learning in Adversarial Settings. In 2016 IEEE European Symposium on Security and Privacy (EuroS&P) , pages 372-387, Mar. 2016. doi: 10.1109/EuroSP.2016.36.

F. Rellich and J. Berkowitz. Perturbation Theory of Eigenvalue Problems . New York University. Institute of Mathematical Sciences. Gordon and Breach, 1969. ISBN 9780677006802.

M. T. Ribeiro, S. Singh, and C. Guestrin. " why should i trust you?" explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining , pages 1135-1144, 2016.

S. Salman, M. M. B. Shams, and X. Liu. Intriguing equivalence structures of the embedding 376 space of vision transformers, 2024. arXiv:2401.15568. 377

378

379

380

M. E. Taylor. Partial differential equations. I: Basic theory , volume 115 of Appl. Math. Sci. Cham: Springer, 3rd corrected and expanded edition edition, 2023. ISBN 978-3-031-33858-8; 978-3-031-33861-8; 978-3-031-33859-5. doi: 10.1007/978-3-031-33859-5.

381

382

383

384

385

386

387

388

389

390

391

392

C. Toraman, F. ¸ Sahinuç, and E. H. Yilmaz. Large-scale hate speech detection with crossdomain transfer. In Proceedings of the Language Resources and Evaluation Conference , pages 2215-2225, Marseille, France, June 2022. European Language Resources Association. URL https://aclanthology.org/2022.lrec-1.238 .

L. N. Trefethen and D. I. Bau. Numerical linear algebra. Twenty-fifth anniversary edition , volume 181 of Other Titles Appl. Math. Philadelphia, PA: Society for Industrial and Applied Mathematics (SIAM), 2022. ISBN 978-1-61197-715-8.

M. G. Vilas, T. Schaumlöffel, and G. Roig. Analyzing vision transformers for image classification in class embedding space. Advances in Neural Information Processing Systems , 36, 2024.

M. Wu, H. Wu, and C. Barrett. Verix: Towards verified explainability of deep neural networks. Advances in neural information processing systems , 36, 2024.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: In the abstract and introduction we claim that we present a method for the exploration of equivalence classes in the input space of Transformer models, which is analyzed in depth in Section 3. The mathematical theory we refer to is deepened in Section 2.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss the limitations and tradeoff given by numeric integration in Subsection 3.1 and theoretical assumptions are enumerated in Section 2. Computational efficiency of our algorithms is discussed in Subsection 3.1. We conducted experiments on 3 datasets only, one of which of small dimensions since our main focus is on the mathematical theory grounding the application of the method to Transformers, as stated at the beginning of Section 4. Other limitations are mentioned throughout Section 4, including the fact that our investigations in the human-readable scenario are at a preliminary stage.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best

judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: The full proofs are part of two previously published papers which we cannot disclose for anonymity requirements. We replicate the relevant proofs in the supplementary material, part of which will be removed from the final version of the paper, referencing to the other papers.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Pseudo-code of the proposed algorithms is reported in Subsections 3.1 and 3.2 so to make the algorithms reproducible, plus our implementation is made available in the supplementary material. Experiments, including the complete setting, and the respective baselines are described in Section 4.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: All experiments are made reproducible through scripts provided as supplementary material.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: All details are provided in Section 4: details of the analyzed architectures, number of iterations of the SiMEC/SiMExp algorithms, technical infrastructure on which the experiments were performed, amount of data the experiments were performed on.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: The standard deviation is reported for the experiment that supports the claims of the paper, i.e. the one on feature importance-based explanations, in Section 4. Standard deviation is also reported for the number of uninformative images produced by the perturbation-based baseline method in the Input space exploration experiment.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: All the experiments were performed on the same infrastructure, which is reported in a footnote in Section 4. Time of execution is one of the key indicators reported for the Input space exploration experiments. More computing power would be required for experiments on bigger Transformer models.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We comply with the terms of use of the datasets employed in the experiments, and we deem our work has no potentially harmful effect on people safety, security, discrimination, surveillance, harassment, nor on human rights. Our proposal does not contribute to spread bias and unfairness towards certain groups of people nor to harm the environment.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [No]

Justification: Although the impacts of XAI on society is broad and deep, in this paper we focus only on the technical problem of exploring the equivalence classes in the input space of Transformers, which doesn't add any specific impact to the discussion about XAI in general.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The datasets used in the paper are explicitely mentioned in the references, as required by the terms of use. Where applicable, the license is also reported.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: Our work doesn't include crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

711 712 713 714 715 716 717 718 719 720 721 722 723 724 725 726 727 728 729

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: Our work does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a mathematically rigorous methodology based on Riemannian geometry for attributing network importance of tokens in a transformer models input space (e.g. image patches, or ~words in the textual domain). The proposed methodology—whilst based on sound theory—translates into an intuitive algorithm involving what appears to be a relatively inexpensive eigendecomposition. Experiments on 3 datasets across both the image and NLP domains explore how the features correlate with ground-truth inputs in the text domain, in addition to first steps towards exploring how the features affect the networks’ output logits.\n\nSTRENGTHS:\n- A major strength of the paper is the mathematically solid approach in attempting to identify regions of the **input** space that explain transformers’ model decisions. This is an important area of study: in contrast to many recent mechanistic interpretability methods finding latent network representations (that are intrinsically hard for humans to interpret by default), salient features in the pixel/text input space are much more readily interpreted by humans.\n- Whilst I am unfamiliar with geometric deep learning, the authors do a fantastic job of presenting the technical content in a digestible manner without sacrificing depth or rigor.\n\nWEAKNESSES:\n# [W1] Feature importance comparisons\n\nFeature importance-based explanations are motivated on [L303] as quantifying the contribution of features `\"to a model prediction\"`. More concretely, around [L180], the authors motivate the eigenvalues of the pullback metric found using their method as ultimately deducing the importance of each segment (e.g. image patch)  `“with respect to the the final prediction”`. \n\nConsequently, a major weakness of the paper is how there is no comparison with related work for how well the proposed method’s identified important features alter the **output** logits (e.g. upon ablation).\n\nI am slightly confused by why the authors did not adopt the established “perturbation test” experimental protocol in the baseline [1] against which they compared, to provide experimental evidence in favor of this. Currently, the only comparisons made around [L274] measure the features’ importance as they correlate to the *input’s* labels.\n\nConcretely, the authors could, for example, ablate particular patches of MNIST and observe that the resulting performance drops correlate with the pullback metric’s eigenvalues. This would provide stronger evidence of the authors’ claims about the features affecting the networks’ output, and (crucially) ground the results in contrast to those achievable by existing methods.\n\n# [W2] Limited experimental results & improvements\n\nThere is a lot of interesting theory here, but ultimately this is a paper with a concrete applied goal of feature attribution in transformer models. With such a new methodology with many technical details, I believe there is an extra burden of proof on the authors to demonstrate this somehow leads to additional insights / practical gains. As such, it is a relative weakness of the paper that so few experiments are performed to justify the methodology.\n\nBeyond toy datasets, it would be interesting to see how the method performs on more complex ones (not necessarily larger ones), such as TinyImageNET. Here, we could visualize much more easily if the method helps identify salient features of animals’ body parts (for example) as being important features for classification. MNIST experiments alone in the image domain are hard to interpret given the similarity of all the input data. \n\nFurthermore, the method provides an almost insignificant increase of just `0.07` cosine similarity (over the baseline in [1]), on just a single dataset (and with just two baselines—for example, how does GradCAM perform here?). This is not sufficient evidence to convince me as a reader that the proposed methodology should be adopted. \n\n---\n\n- [1]: Chefer, Hila et al. “Generic Attention-model Explainability for Interpreting Bi-Modal and Encoder-Decoder Transformers.” 2021 IEEE/CVF International Conference on Computer Vision (ICCV) (2021): 387-396.\n\nQUESTIONS:\nNo additional questions at this time (beyond those alluded to in the weaknesses section).\n\nLIMITATIONS:\nSome limitations are indeed addressed throughout. However, (unless I have missed something, in which case I apologise!) I can only find the limitations of the small number of datasets used stated in the NeurIPS checklist. This needs to be stated explicitly in the main paper.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work attempts to find the set of inputs that generate the same neural network predictions. To this end, the authors interpret the layers of the network as transformations of the input manifold. This interpretation is used to defined equivalence classes over the inputs and to define feature importance. Finally, the tools are used to identify equivalence classes for MNIST digits and for hate speech detection with BERT.\n\nSTRENGTHS:\nSection 2 does a thorough job of introducing a manifold interpretation to neural networks.  This introduction is then used to motivate multiple algorithms for finding equivalence classes of the inputs---or the setup of inputs that result in the same prediction---and identify features that are important.\n\nWEAKNESSES:\nThe main contribution of this work is to introduce a tool for analyzing which set of inputs produce the same output. However, this is exactly the Fisher information matrix (with respect to the inputs) and has also been introduced in prior work (https://arxiv.org/abs/2104.13289). Could the authors clarify what the differences are and what the additional novelty. If the \"local data matrix\" introduced in https://arxiv.org/abs/2104.13289 is identical to the tools in this work, I think it severely diminishes the contributions of this work. Furthermore the experiments are extremely similar (such as Figure 1). \n\nThe second weakness is the limited number of experiments. The work does not show any quantitative results: Figure 1, Figure 2 and Figure 3 is just 1 example and is not indicative of why the tools are useful. The experiments in section 4 primarily discuss wall-clock time. It would significantly help if claims such as \"(Line 266) we notice that the perturbation-based algorithm ends up producing monochrome ...\" are substantiated quantitatively. Overall, the work doesn't provide novel tools and the experiments lack a novel usage of these tools and do not reveal any new insights.\n\nQUESTIONS:\n1. How does this work differ from https://arxiv.org/abs/2104.13289?  \n2. What is algorithm 4 (exploration) used for? \n3. Many experimental details are lacking. It would help to add appendices clarifying what networks were used and how they were trained. Overall, the work feels incomplete in many respects\n4. For feature importance-based explanations, there are many other methods like LIME/SHAP (and many more follow up papers) which the authors do not compare to. Why were they omitted?\n5. Does this method scale to larger datasets, considering we have to compute outer-product of the gradients and compute the Eigenvalues of this matrix?\n\nLIMITATIONS:\nThe authors address limitations of their work but it can be expanded upon. For example, the authors can discuss the time required to compute Eigenvalues, and other limitations such as not having any Eigenvalues to be 0 in Algorithms 3 / 4. Furthermore, their algorithm should work (in theory) for infinitesimal steps in the input manifold.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-16"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors present a method for exploring equivalence classes in the input space of Transformer models using a solid mathematical theory. By analyzing the Jacobian of the model, the method reconstructs and navigates these classes, offering a powerful tool for understanding Transformer interpretations and enhancing explainability. The proposed method is expected to solve problems in Computer Vision and Natural Language Processing tasks.\n\nSTRENGTHS:\nI write both strength and weakness. \n\nFirst, I must disclose that I have no prior study or background in both Transformers and manifolds. While I am conceptually aware of them, my knowledge is limited to that extent, and I lack confidence in reviewing the technical details. Therefore, please consider my review comments as feedback from a layperson in this field, focusing on the overall mathematical consistency and readability of the paper.\n\nThis paper describes mathematics in a clear and understandable manner that even a layperson like myself can grasp. Each definition and theorem is stated accurately, and I believe that the general concepts can be understood with basic knowledge.\n\nI personally feel that the objective of this paper is not clearly conveyed. While the paper claims to contribute to explainability and sensitivity analysis through the analysis of input manifolds, the logic behind this was not clear to me in the Introduction and Preliminaries. Although the concepts of explainability and sensitivity analysis become clearer in the later chapters, it might be beneficial to provide a bit more explanation in the Introduction.\n\nAdditionally, it might be helpful to clearly define the equivalence class mathematically.\n\nSince I am not familiar with the existing literature, I was unable to judge the novelty of this work.\n\nWEAKNESSES:\nSee above.\n\nQUESTIONS:\nSee above.\n\nLIMITATIONS:\nN/A.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-19"}

comments: SUMMARY:
This paper develops a novel theoretical framework grounded in Riemannian geometry for analyzing the input space of Transformer models, and introduce two algorithms, SiMEC and SiMExp, which facilitate the exploration and interpretation of equivalence classes within this input space. These methods offer new insights the internal mechanisms of Transformers, and provide new understanding of how these models perceive and process input data which can be very useful in the field of explainable AI.

STRENGTHS:
1 novelty: This paper provide an innovative application of Riemannian geometry to analyze the input spaces of Transformer models, which is very novel in the area.

2 Theory: This paper establishes a solid mathematical theory on how Riemannian geometry is applied to Transformer models. Based on this theory, SiMEC and SiMExp are developed to explore the input spaces of Transformer models.

WEAKNESSES:
In experiment, the MNIST dataset is a little bit trivial, as the pixels of the background is essentially zero. It is nice to see the application of the proposed algorithm on natural images like CIFAR.

QUESTIONS:
See above

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-28