DORA: Exploring Outlier Representations in Deep Neural Networks
Authors: Kirill Bykov
Mayukh Deb
Dennis Grinwald
Klaus Robert Muller
Marina MC Höhne
Venue: Accepted by TMLR ()

Abstract: Deep Neural Networks (DNNs) excel at learning complex abstractions within their internal representations. However, the concepts they learn remain opaque, a problem that becomes particularly acute when models unintentionally learn spurious correlations. In this work, we present DORA (Data-agnOstic Representation Analysis), the first data-agnostic framework for analyzing the representational space of DNNs. Central to our framework is the proposed Extreme-Activation (EA) distance measure, which assesses similarities between representations by analyzing their activation patterns on data points that cause the highest level of activation. As spurious correlations often manifest in features of data that are anomalous to the desired task, such as watermarks or artifacts, we demonstrate that internal representations capable of detecting such artifactual concepts can be found by analyzing relationships within neural representations. We validate the EA metric quantitatively, demonstrating its effectiveness both in controlled scenarios and real-world applications. Finally, we provide practical examples from popular Computer Vision models to illustrate that representations identified as outliers using the EA metric often correspond to undesired and spurious concepts.

Published in Transactions on Machine Learning Research (06/2023)
DORA: Exploring Outlier Representations in Deep Neural Networks
Kirill Bykov
kbykov@atb- wwwpttttt-potammmmmmm
Understandable Machine Intelligence Lab Leibniz Institute for Agriculture and Bioeconomy (ATB), Potsdom, Germany Technicol Universily of Berlin, Berlin, Germany
Mayukh Deb
mappdammaiinnk33000@mmaaaaaa
Independent
Dennis Grinwald
demnis.grinuuuldBBu.bbl...
Machine Learning Group
Technicul Universiby of Berlin, Berlin, Germany
BIFOLD Berlin Institute for the Foundations of Learning and Data, Berlin, Germany
Klaus-Robert Müller
Hluus-roder..muullerr@uu
Machine Learning Group
Technical University of Berlin, Berlin, Germany
BIFOLD Berlin Institute, for the Foundations of Learning and Data, Berlin, Germany Department of Artificial Intelligence, Korea University, Scoul 136-713, Korea Max Planck Institut für Informatik, 66123 Saarbrücken, Germany Google Research, Brain Team, Berlin, Germany
Marina M.-C. Höhne
mbochne@@ll-polleee de
Understandable Machine Intelligence Lab
Lešbniz Institute for Agriculture and Bioeconomy (ATB), Potsdam, Germang Deportment of Computer Science, University of Potsdam Department of Physies and Technology, UTT Arctic University of Norway
BIFOLD Berlin Instibule for the Foundations of Learning and Data, Berlin, Germany
Reviewed on OpenReview: https: // openreview. net/ formm? tdd.uumueeiyyeueeeeee
Abstract
Deep Neural Networks (DNNs) excel at learning complex abstractions within their internal representations. However, the concepts they learn remain opaque, a problem that becomes particularly acute when models unintentionally learn spurious correlations. In this work, we present DORA (Data-agnOstic Representation Analysis), the first data-agnostic frame- work for analyzing the representational space of DNNs. Central our framework the proposed Andreme-Activattion (EA) distance measure, which assesses similarities between representations by analyzing their activation patterns data points that cause the highest level of activation. As spurious correlations often manifest in features of data that are anomalous to the desired task, such as watermarks or artifacts, we demonstrate that internal representations capable of detecting such artifactual concepts can be found by analyzing relationships within neural representations. We validate the EA metric quantitatively, demon- strating its offectiveness both in controlled scenarios and real-world applications. Finally, we provide practical examples from popular Computer Vision models illustrate that representations identified outliers using the EA metric often correspond to undesired and spurious concepts.

---

Published in Transactions on Machine Learning Research (06/2023)
1 Introduction
The ability of Deep Neural Networks (DNNs) to perform complex tasks and achieve state-of-the-ort performance in various fields can be attributed to the rich and hierarchical representations that they learn Bengio el. al. (2013). Far beyond the handcrafted features that were inductively imposed by humans on learning machines in classical Machine Learning methods Marr and Nishihara (1978) Jackson Fogel and Sagi (1989). Deep Learning approaches exploit the network's freedom to learn complex abstractions. However, a prevalent concern remains that the nature of concepts, learned by the model remains unknown. The rapid progress representation learning only exacerbates the issue of interpretability, since DNNS are frequently trained using self-supervised methodologies JJiswal et al. (2020); LeCun and Misra (2021) on immense volumes of data
Brown et al. ) Bommasani et al. [2021] which accelerated the umpredictability concerning the scope of possible learned concepts and their mutual relations Goh et al. (2021)
The increasing popularity of Deep Learning techniques across various fields, coupled with the difficulty of interpreting the preision-making processes of complex models, has led to the emergence of the field of Explainable AI (XAT) (c.g. Montavon al. (2018): [Samck et al. (2019); Xu et al. (2019); Gade ct al. (2019); Rudin )019); Samek et al. (2021)). Research within XAI has revealed that the internal representations
that form the basis of DNNs are susceptible to learning harmful and undesired concepts, such as biases Guidotti et al. (2018); ; Jiang and Nachum (2020), Clever Hans (CH) effects Lapuschkin et al. (2019) and
backdoors Anders et al. (2022). These malicious concepts often are unnatural or anomalous in relation to the relevant concepts within the dataset. Examples include watermarks in the PASCAL 2007 image classífication
task Lapuschkin et al. 2009), Chinese logographic watermarks in ImageNet dataset Li et al. (022). colored
band-aids in skin-cancer detection problem Auders et al. (2022) or tokens in a pneumonia detection problem
Zech et al. (2018).
To enhance our understanding of the decision-making processes within complex machines and to prevent biased potentially harmful decisions, it is crucial to explain the concepts learned during training. By analyzing the relationships between internal neural representations, we can gain insights into the model's predictive strategics. In this work, we introduce Representation Analysis, a framework dedicated to exploring the representations of a particular model layer. Our approach utilizes a proposed which measures the similarity between various learned representations within the networks by examining Signals (AMS). These signals represent data
(EA) metric,
the common activation patterns on points where the representations exhibit their highest activations and can be identified through either a dala-aware (natural) process from an existing data corpus [Borowski el al. (2020) or a dala-agnoslic process, in which the signals are synthetically generated Erhan et al. (2009); Olah et al. (017)) We refer to the representation analysis conducted with the latter method as DORA" (Data-agnOstic Representation Analysis). We demonstrate the interpretability of our proposed distance measure and study the connections between Signals. Moreover, we quantitatively assess the alignment.
natural and synthetic Activation-Maximisaaaa between the functional EA distance and human perception we demonstrate that EA distances between representations generally align with human judgment regarding the similarity of concepts, particularly in scenarios where the concepts underlying the representations are known. Additionally, we highlight proposed distance measure's 's ability to establish robust bascline for detecting inserted anomalous concepts in controlled scenarios. Lastly, through practical experiments conducted on popular Computer Vision models, we reveal that anomalous representations identified by our framework often correspond to undesirable spurious concepts.
2 Related Work
To address the concerns regarding the black-box nature of complex learning machines Baehrens et al. 2010
Vidovic et al. (2015): Buhrmester et al. (2019); Samek et al. (002)), the field of Explainable Al (XAI) has emerged. While some recent rescarch focuses on inducing the self sxpl-ixp capabilities through changes in
the architecture ure and the learning process Gautam el al. (2022a bi: Chen et al. (2018); Gautan el al. 2021
the majority of XAT methods (typically referred to as post-hoe explanation methods) are decoupled from the
TPyTorc implementation of the proposed method can be found by the following link:
2

---

Published in Transactions on Machine Learning Research (06/2023)
training procedure. A dichotomy of post-hoc explanation methods could be performed based the scope of their explanations, i.e., the model behavior can be either explained on local level, where the decision-making strategy of a system is explained for one particular imput sample, or on a global level, where the aim is to explain the prediction strategy Iearned by the machine across the population and investigate the purpose of individual components in a universal fashion detached from single datapoints (similar to feature selection Guyon and Elisscoff (2003))-
Local explanation methods typically interpret the prediction by attributing relevance scores to the features of the imput signal, highlighting the influential characteristics that affected the prediction the most. Various methods, such as Layer-wise Relevance Propagation (LRP) Bach et al. (2015), GradCAM [Sclvarauu et al.
(201)) Occlusion Zeiler and Fergus (2014), MF1 Vidovie et al. (2016), Integrated Gradient Sundararajan et al. 2017p. have proven effective in explaining Graph Neural Networks Wang et al. (001 Tiddi et al.
(2020) as well as Bayesian Neural Networks Bykov et al. (2021) Brown and Talbort (2022). To further boost the quality of interpretations, several enhancing techniques were introduced, such as SmoothGrad Smilkov jet al. (2017] Omeiza et. al. (2019). NoiseGrad and FusionGrad BBykov et. al. (2022). Considerable attention also has been paid to analyzing and evaluating the quality of local explanation methods (e.g. [Samek et al. (2016);
; Hedström et al. 20022 Guidotti (2021): Binder et al. (2023)) However, while the local explanation paradigm is incredibly powerful in explaining the decision-making strategies for a particular data sample, the main limitation of such methods is their inability to effectively investigate the unexplored behaviors of the models, such as the detection of previously unknown spurious correlations and computational shortcuts Adebayo et al. (2022).
Global explanation methods aim to interpret the general behavior of learning machines by investigating the role of particular components, such as neurons, channels, or output logits, which we refer to representations. Existing methods mainly aim to connect internal representations to human understandable concepts, making the purpose and semantics of particular network sub-function transparent to humans. Methods such as Network Dissection Bau et al. (2017: 2018) and Compositional Explanations of Neurons Mu and Andreas (2020) aim to associale representations with human--understanllbbee concepts. They achieve this by examining the intersection between the concept-relevant information provided by binary mask and the activation map of the corresponding reprosentation. The MILAN method Hernandez et al. (2021) generates text description of the representation by searching for text string that maximizes the mutual information with the image regions in which the neuron is active.
2.1 cotivaation-xximisaatn Methods
The family of Activation Maximization (AM) Erhan et al. (2009) methods aims to globally explain the concepts behind neurons by identifying the input that triggers maximal activation in a particular neuron network layer, thereby visualizing the features learned. These inputs, which we will refer to as Activation- Maximization Signals (AMS), could be either natural, found in a dala-aware fashion by selecting a "real" example from an existing data corpus Borowski et al. (2020), or artificial, found in a data- aato-agnostic mode by generating a syuthetic input through optimization Erhan et al. (2009): Olah et al. (017) Szegedy el al. (2013).
In comparison to carlier synthetic AM methods, Feature Visualization (FV) Olah et al. oplimization in the frequency domain by parametrizing the image with frequencies obtained from the Fourier
2017 performs
transformation. This reduces adversarial noise in resulting explanations (e.g. Erhan et al. (2009); Szegedy et al. (2013) improving the interpretability of the obtained signals. Additionally, the FV method applics multiple stochastic image transformations, such as jittering, rotating, or scaling, before each optimization step, as well as frequency penalization, which either explicitly penalizes the variance between neighboring pixels or applies bilateral filters on the input.
2.2 Spurious Correlations
Deep Neural Networks are prone to learn spurious representations - patterns that are correlated with a target class on the training data but not inherently relevant the learning problem Izmailov et al. (2022). Reliance spurious features prevents the model from generalizing, which subsequently leads to poor performance
3

---

Published in Transactions on Machine Learning Research (06/2023)
sub-groups of the data where the spurious correlation is absent (cf. Lapuschkin et al. 2016 2019 Geirhos
et al. (2020)) In the field of Computer Vision, such behavior could be characterized by the model ' reliance
on aspects such as an images background Xiao et al. (2020), object textures Geirhos et al. (2018). or the
presence of semantic artifacts in the training data Wallis and Buvat (2022) Lapuschkin et al. (019)) Geirhos
et al.
2020j; Anders et al. (2022). Artifacts can be added to the training data on purpose as Backdoor
attacks Gu al. (2017); (Tran et al. (2018). or emerge naturally and might persist unnoticed in the training
corpus, resulting Clever Hans e/fects Lapuschkin al. (2019).
Recently, XAI methods have demonstrated their potential in revealing the underlying mechanisms of predictions made by models, particularly in the presence of artifacts such as Clever Hans or Backdoor artifacts. Spectral Relevance analysis (SpRAy) aims to provide a global explanation of the model by analyzing local explanations across the dataset and clustering them for manual inspection Lapuschkin et al.] (2019). While successful in certain cases [Schramowski et al. (2020), SpRAy requires a substantial amount of human supervision and may not detoct artifacts that do not exhibit consistent shape and position in the original
images. SpRAY-based Class Artifact. Compensation Anners et. al. (2022) method allowed for less human supervision and demonstrated its capability to suppress the artifactual behavior of DNNs.
2.3 Comparison of Representations
The study of representation similarity in DNN architectures is topic of active research. Numerous methods comparing network representations have been applied to different architectures, including Neural Networks of varying width and depth Nguyen et al. (2020), Bayesian Neural Networks Grinwald et al. 2022 and
Transformer Neural Networks Raghu et al. 22021). Some works Ramsay et al. (1984); Laakso (2000 Kornblith
ct al.
(019 Nguyen et al. (2022) argue that the representation similarity should be based on the correlation
of a
distance measure applied to layer activations on training data. Other works Raghu et al.
(2017); Morcos
et al.
2018
compute similarity values by applying variants of Canonical Correlation Analysis (CCA) Hardoon
et al.
(2005 : Bießmann et al. (2010) on the activations or by calculating mutual information Li et al.
(015)),
employ kernel methods to quantify the evolution of the representations Montavon et al. (2011) Braun
et al. spaces, e.g. layers, and not individual components, and as a result often overlook the semantics of learned concepts. Furthermore, those methods are dependent on the availability of data.
22008). However, these methods are predominantly utilized for the comparison of whole representation
3 Distance Metrics between Neural Representations
In the following, we start with the definition of a neural representation as a sub-function of a given network that depicts the computation graph, from the input of the model to the output of a specific neuron. Definition (Neural representation). We define neural representation f as a real-valued function f - IR, mapping from the data domain D to the real numbers R.
The following definition is introduced to highlight the distinction between the traditional notion of neuron and the broader computational process encapsulated in the term neural representation. Conventionally, neuron is defined as a function that takes inputs from its preceding neurons. In contrast, a neural representation describes the entire computational process, starting from the input and yielding the activation of a specific neuron (unit). While some neurons in DNNs produce multidimmnsioonal outputs, depending on the specific use cases, multidimensional functions could be regarded her as a set of individual representations or alternatively could be aggregated to achieve scalar output. For example, the case of convolutional neurons that output activation maps containing the dot product between filter weights and input data each location, activation maps could be aggregated by average- or max-pool operations for the sake of simplifying the explanation of the semantic concept underlying the function. The choice depends on the particular aim and scope of the analysis and does not alter the network itself.
The scalar output of representations often corresponds to the amount of evidence or similarity between concepts present in the input and internally learned abstractions. Varions sub-functions within the model could be considered as neural representations, ranging from the neurons in the initial layers that are often regarded elementary edge or color detectors Le and Kayal (2021), to the model output. Throughout this
4

---

Published in Transactions on Machine Learning Research (06/2023)
work, primarily focus on the high-level abstractions that emerge in the latest layers of networks, such the feature-extractor layers in well-known Computer Vision architectures, as they are frequently employed for transfer learning Zhuang et al. (2020).
In DNNs, neural representations are combined into layers - collections of individual neural representations that typically share the same computational architecture and learn abstractions of similar complexity. In the scope of the following work, we mainly focused on the analysis of the relations between representations within one selected layer from the network.
Definition 2 (Layer). We define loyer F = [[) set comprising k inndvvidual neural represenla- tions.
To examine the relationships between representations, we begin by analyzing the behavior of functions with respect to given dataset. We define dataset, D, consisting of N data points denoted as D = { ... uN This set is referred to as the evaluation dataset and is used to measure the relationship between two neural representations. We make the assumption that these data points, II,...,IN, are independently and identically distributed (i.i.d.) samples from the overall data distribution D. Additionally, we standardize the activations of representations on this evaluation dataset, resulting a mean of 0 and a standard deviation 1.
For a neural representation ft and an evaluation dataset D "el,,NNN we define a vector of activations
([i(x)),...,,ii
(1)
where
1 N N
N N Σ (f((z) -4) =1 =1
H4
Σ f((z)) =0, oz
(2)
Standardizing the vectors in this way can help to mitigate any differences in scale between the vector components and ensure that each component contributes equally to the distance calculation. Below, we present three widely recognized metrics that can be utilized to measure the distance between neural representations.
Minkowski distance:
N dM (f) = Σ f5(e))
(3)
where 2 1,p E E determines the degree of the norm, which gauges the sensitivity of the metric to differences between the components of the vectors being compared. In general, larger values of lead to a greater emphasis on larger differences between the components the vectors. Conversely, smaller values of p reduce the influence of larger differences, leveraging an increased uniform weighting of all components.
Pearson distance:
dp (fr. V2 V Pp (a, ay),
where (p b) is the Pearson correlation coefficient between the vectors and b. The Pearson correlation coefficient is a widely used metric for measuring the linear dependence between two random variables. is an interpretable mcasure of similarity, however, it is also sensitive to outliers, which can significantly affect the calculated distance.
Spcarman distance:
ds (f) V2 1 1- a;),
55)
where (a, b) is the Spearman rank- cor-eoorel coefficient between vectors a and b. The Spearman correlation non-parametrie rank-basod metric commonly used to measure the monotonic depen- dence between two random variables. Its main advantage is that is robust to outliers and can handle ties in the data.
5

---

Published in Transactions on Machine Learning Research (06/2023)
0.5
0.4
Densky
0.3
0.2
0.1
0.0
-2
2
ResNe: 18''iirrrann husky' activation
Figure 1: Distribution of activations for the "Siberian husky" representation. From the figure we can observe the standardized activation distribution of the "Siberian husky" logit from the ResNet18 model trained ImageNet. The data was collected across the ILSVRC-2012 validation dataset. Additionally, various input images are dispalyed over their respective activations. Analyzing the activation of representations can provide crucial insights into the behavior of the model. For instance, we observe that the model achieves extremely high activations when there are multiple dogs in the image, corresponding to the "Dogsled" class. However, we also observe a potential spurious correlation, where the model assigns high scores to images with a snowy background.
3.1 Data-Aware -cttvme-cttvatiin distance
The meaning and semantics of individual representations are frequently characterized and explained by datapoints where these representations exhibit extreme values. This analysis typically concentrates on the most positively activating signals, primarily duc to the prevalent use of bounded activation functions like ReLU Glorot et al. (2011). In these functions, positive activation values typically signily the existence of specific patterns within the input signal.
Given the evaluation dataset D and neural representation F, we define collection of natural Activation-Maximisttion signals (n-AMS) as follows:
Definition (n-AMS). Let f; E F be neural representation, and D (ai, END C be an evaluation dataset with N datopoints. Assume that the dataset could be split in n disjoint blocks D U Dr, D.
(Bu+th-.. , (O...., 1 of length d.
We define collection of n natural Aetiiatton-MMaiimisiii
signals (n-AMS) as S (sf "i,, 3 where
arg max (x), Vt {0, (0,...,,.. 11.
(6)
DE
The suggested definition of n-AMS diverges from the conventional method for explaining the concepts behind the representation by analyzing signals with the highest activations, that is, signals whose activations rank highest across the entire dataset. Instead, n-AMS could be seen samples from the Extreme Value distribution, thereby allowing for statistical properties to be considered. Additionally, such ditribution is parameterized with parameter d, referred to the depth, which represents the size of the subset from which the signal is obtained. Note that we could examine the highest activation signal of the whole datasct by setting n = 1 and d = N, however, interpreting the representation's 'ss semantics by using only one signal might be misleading. Figure illustrates the distribution activations of the "Siberian husky" logit from the ResNet18 model trained on ImageNet He et al. (2016) across all the images from the ILSVRC-2012 validation dataset Russakovsky et al. (2015), where we can observe that the most activating signal corresponds to the a "Dogsled" class. In light of this, aim to sample several n-AMS from separate data subsets.

---

Published in Transactions on Machine Learning Research (06/2023)
We propose that by examining how two neural representations activate each other's n-AMS, gain significant insights into the similarity of the learned abstractions. For this, we first introduce the representation activation vectors (RAVs).
Definition 4. Let F iff..... f be layer including l. neural representations, and S ........ S be collection of n. N-AMS for each of the k represenlations in the layer. For Vi,j E {1,...,k we define n3 - (s)) as mean activation of fa given the n-AMS of fs.
For any two representations fi, E F,, we define their pair-wise representation activation vectors (RAVs) Vija jj as:
14 (1)
rij
Tjí
In addition, for each neural represenlation Jt E F, we define the corvesponding layer-wise RAV as follows:
T'
(8)
Intuitively, the idea behind RAVs is to capture how one representation's n-AMS are perceived by other representations. RAVs capture the direction of n-AMS signals within two-dimensional vectors when dealing with pair-wise vectors, encoding the information about how two neural representations respond to each other's stimuli. In the layer-wise case, the vectors are k-dimensional, utilizing all representations within the layer as descriptors. In practice, to compute RAVs, n n-AMS are gathered for each representation within the layer, inferenced by the model. Then, activations across the representations from the layer collected and averaged.
To illustrate the concept of Representation Activation Vectors, we calculated n-AMS for five distinct neural representations extracted from the output layer of the ImageNet pre-trained ResNet18 model. These representations corresponded to the classes "Siberian husky", " "Alaskan malamute", "Samoyed", "Tiger cat", and " "Aircraft carrier", which were selected manually to demonstrate the decreasing visual similarity between the classes and the "Siberian husky" class, Using the ILSVRC-2012 validation dataset, we computed the signals with sample size of 100 and subset size of d 500. Figure presents scatter plot of aclivation values across dalapoints and pair-wise RAVs. Our results indicate that the angle between these vectors increases with the visual dissimilarity between the classes.
To measure the distance between neural representations that reflect the similarity of learned concepts, we introduce novel distance metric known as the similarity between two neural representations based on the angle between their Representation Activation Vectors. Given that the computation of this distance measure is performed in dala-aware mode and relies the presence of dataset, we refer to this distance the natural Activvaiot distance EA,-
distance. This metric assesses the
Definition (Fxtremme-Attivation Activation distance). Let fú, JJ E F be two neural represenlations, and rajs Tjs be their wise RAVs. We define poir-wise Extreme-Activatio Activation distonce as
d LeA (fr,f)
V2 cos (rig.
(9)
where cos(A, B) is the cosine of the angle between vectors A, B. Additionally, we define layer-wise Entreme-Activation distance between ft. fy as
1
dEA. ff)
V2 V 1 cos (rin, T.
(10)

---

Published in Transactions on Machine Learning Research (06/2023)
RocNet 10, Samoyed
Barroyed
Reshae 18, Tiger oar
18/Arrreet
+
O N
Reetel
Hulky
Husky
Hassyy
ResNet 10, Siberan husky
ResNet 18, Siberian husky
ResNet 18, Siberian husky
ReeNet 10, husky
Figure Joint activation and pair-wise RAVs of different ImageNet representations. Four scatter plots illustrate the joint activations, n-AMS sets, and pair-wise RAVs for distinct pairs of neural representations. Blue points represent the n-AMS of the representation depicted the vertical axis, while orange points represent the n-AMS of the representation shown the horizontal axis. The representations, taken from the ResNet18 output logit layer, include "Alaskan malamute", "Samoyed", "Tiger cat", and "Aircraft carrier", each compared with the "Siberian husky" representation. We can observe that the angle between RAVs reflects the visual similarity between classes, representations were trained to learn: the RAVs of "Siberian husky" and "Alaskan malamute" are almost collinear due to the high visual similarity between the two dog breeds, while the RAVs of "Siberian husky" and " "Aircraft carrier" are orthogonal, indicating their visual dissimilarity.
3.2 Synthetic Extreme-Activation distance
Although data-aware distance metrics can offer insight into the relationships between representations, their dependence on the data can be viewed as a limitation potentially acting as bottleneck when analyzing the relationships between a model's internal reprosentations. Modern machine learning models are often trained on closed-source or very large datasets, making difficult to obtain the exact dataset the model was trained on. If the evaluation dataset, i.c. the dataset utilized for n-AMS sampling lacks concepts that were present in the training data, the resulting n-AMS could potentially be misleading. This is due to the fact that they might not encapsulate the features that the representation has learned to detect, simply because such features are absent in the dataset.
To alleviate the dependence on data, we propose a dala-agnostic method for computing the Extreme-Activatiio distance. This approach employs synthetic Activation- Maximization Aetivaaion-Maxiiization signals, denoted as s-AMS, in place of n-AMS. The S-MMS signals are generated by the model itself through an optimization process, thereby climinating the need for external generative models or datasets.
Definition 6 (s-AMS). Let f: E F be neural representation. Synthetic Activation-Marimiaaaae (s-AMS) signal 5 is defoned a solution to the following oplimization problem:
max S((8), SEA
(11)
where denotes the set of potential solutions, typically defined by the particular signal parametrization employed for optimization.
Generating A-S for neural representation is a non-convex optimization problem Nguyen al. (2019 that typically employs gradient- cradient-based methods Erhan al. 2009): Nguyen et al. (2015); Olah et al. [017 Starting from random noise parametrization of input signals, the gradient-ascend procedure searches for the optimal set of signal paramcters that maximize the activation of a given representation. Early methods employed standard pixel parametrization Erhan et al. (2009), while modern approaches used Generative Adversarial Network (GAN) generators Nguyen et al. (2016) or Compositional Pattern Producing Networks (CPPNs) Mordvintsev et al. (2018); Stanley (2007). In this study, we use the Feature Visualization method Olah et al. (0077) for s-AMS generation, which parametrizes input signals by frequencies and maps them the pixel domain using Inverse Fast. Fourier Transformation (IFFT). This method is popular for its simplicity
8

---

Published in Transactions on Machine Learning Research (06/2023)
and independence from external generative models, well as for its ability to be human-interpretabbee Olah et al. (2020); Goh et al. (2021); Cammarata et al. (2020).
The optimization procedure for s-AMS generation has several adjustable hyperparameters, inchuding the optimization method and transformations ap-
plied to signals during the procedure. A key param- eter is the number of optimization steps (or epochs) anal-
denoted m. This parameter can be ogous to the parameter d used in n-AMS generation. Since different random initializations in the param- eter space can lead to the convergence of A-M generation into different local solutions, the resulting s-AMS can vary. This variability mirrors that ob- served when sampling n-AMS. Therefore, analogous to Definition 3 for the representation fú, we intro- duce a sct of A-M,, comprised of signals, denoted as S, = (,,
Maci
S-AMS
Maximally Activating Images from ImageNet
Figure 3: Failing to explain "Star Wars" repre- sentation with natural images. Comparison of the s-AMS (left) and most activated images collected from the ImageNet dataset (right) for unit 744 in the last convolutional layer of the CLIP ResNet50 model. Due to the inaccessibility of the training dataset and lack of specific images due to copyright restrictions, natural images do not reveal the underlying concept of the "Star Wars" neuron.
Figure 3 demonstrates the comparison between s- AMS and most activating signals from the ImageNet dataset for one unit in CLIP ResNet 50 network The analysis based solely natural signals leads to erroneous conclusions about the learned concept due to the absence of the true concept in the dataset. As the original training dataset remains undisclosed, explaining the concopts learned by the representation via identifying the most activating images from ImageNet may lead to misinterpretation,, given the absence of "Star Wars"-related images within the ImageNet dataset. In contrast, synthetic Activation-Martte Maximization Signals can depict the learned concepts without any dependency data.
The Synthelic Medreme-AAlivvlii distance, or EA,, is defined in a manner analogous to the EA, distance (Definition 5) with the key distinction being that Representation Activation Vectors are calculated using S-AMS instead of n-AMS. Hence, in contrast to EA,, which evaluates the co-activation of representations based on each other's natural Aetivation-Maximizaai signals, EA, assesses how two representations activate in response to each other's synthetic Activation-Maximiaatie signals, iee, signals generated through an artificial optimization process.
fx) distance). Let F= [fi,..., == fk) be a layer including l neural representations, and $ = (S,...,,Su S.) be collection of s-AMS for each of the h representations in the lager. For j E {1....,k, k} introduce synthetic RAVs, by substituting n-AMS with s-AMS in Definition 4 Pays Fzan pair-wise synthelic RAVs, Fj. layer-wise synthelic RAVs.
Definition (Synthetic Extreme-Activee
We define pair-wise and layer-wise synthetic Estreme-Activation distance between f. and fa as
1 22 V cos Py), deA, (f.
1 V2 V 1 cos (F:.,
(fv
(12)
Notably, FA,, distance is computed using standardized activations. However, due to the data-agnostic nature of the EA, distance, standardization cannot be performed without accessing the evaluation dataset; hence, we use the raw representation' 's activations. Although this could be considered as limitation, since the EA, distance is not shift- invtr-invriannt, we found in our practical experiments that the angles between synthetic and natural RAVs are typically maintained.
Signals were obtained from OpenAI Microscope.
9

---

Published in Transactions on Machine Learning Research (06/2023)
3.3 Properties and Limitations
A key characteristic of the Extreme-Activation distance, applicable both natural and synthetic contexts, in comparison to other metrics, such as Minkowski, Spearman, and Pearson metrics, is that its computation is based the activations of two representations given small subset of AMS, enabling a manual examination of these data points. Such analysis of AMS provides insights into the shared concepts between the two representations. Figure 4 illustrates the procedure of calculating EA, distance between two representations obtained from the logit layer of the ResNet18 network, specifically for the "Zebra" " and "Lionfish" repre- sentations. Examination of the angle belween Representation Activation Vectors (RAVs) reveals a mutual co-activation between the two representations on each other's n-AMS. By analyzing the signals themselves, which are displayed the right section of the figure, we can observe the unique shared concepts between these two classes notably the specific black and white striped pattern exhibited by both animals. Thus, by relying on limited number of anchor data points, the EAA distance facilitates the interpretation of the reasons behind the functional similarity between representations.
n-AMS 'zeera.n.00
6.1,11 f. 11.34 fy 4.53,
1.00
Density
0.75
0.50
Restiet 18, "zebra.n.01"
10-
0.25
Fyx
f,. ResNet 18, 'zebra.n.01"
n-AMS "lionfish.n001
1.00
9.95. f, 4.56
9.65, f 3.66 10.21. 3.89
Density
0.75
0.50
"Zebina n-MS
0.25
Reqular datapoints
0.00
10
10
Reshet 'lonfish.n.01"
'lioffih.nn001
Figure 4: Explaining the Extremo-Activation Distance. Figure illustrates EA, distance between two logit representations from ResNet18 network, corresponding to "Zebra" and "Lionfish" classes. Teft. figure demonstrates that RAVs experience colinearity, implying mutual co-activation on each other n-AMS (n 10, d = 10000). The center part of the figure presents the activation distributions of representations across the evaluation datasel, where the highlighted points correspond to n-AMS. We can observe that both representations are strongly activated by each other's n-AMS, with the "Lionfish" represented in orange and the "Zebra" " in blue. The right-hand portion of the figure presents several n-AMS examples, highlighting the shared feature between the images the black and white stripe pattern present in both animals which explains their mutual co-activation.
EA distance, as discussed above, operates the premise that the representations could be explained by the features in input, that maximally activate them. This viewpoint could be seen as an oversinplification, as it neglects certain attributes that moderately activate or even de-activate representations. Depending on the specific problem and the type of representations question, such nuances could hold significant importance for understanding the internal dccision-making mechanisms of the network. Nonctheless, our experimental findings underline that the EA distance, even when solely focusing on the most activating signals, offers potent and practical framework for interpreting Deep Neural Networks. This is particularly applicable when examining representations with output confined to the positive realm, for instance, post-ReLU activation function.
One limitation common to all distance metrics between neural representations, including the proposed EA distance, their inability to account for the multisemanticity or multimodality Goh al. (2021) of neural representations the capacity of a single representation detect various concepts. the case of the EA distance, this behavior be reflected in the high variance of the RAVs, which results from the fact that.
10

---

Published in Transactions on Machine Learning Research (06/2023)
h
f2
F4
16
5
52
86
44
85
16
Representation Arlas
Outlier Detection
Inspecting Associations
1. Layer selection
2. EA distance computation
3. Analysis
The figure depicts the three fundamental stages of the analysis: 1. Selection of the layer of interest within the specified model. 2. Caleulation of EAA distances between representations. 3. Analysis of the relationships between representations, encompassing the visualization of reprosentation space, identification of outlier reprosentations, and, when necessary, manual scrutiny suspicious relationships between representations.
Figure 5: Outline of the Representation Analysis:
n-AMS originate from multiple modalitics, i.c., different concepts. While this remains an open avenue for future research, our work demonstrates that simply averaging activations for RAVs computation provides an effective distance measure.
4 Representation Analysis
Traditionally, local XAI methods used to analyze model's docision-making process, particularly when assessing potential reliance unwanted spurious concepts. However, these techniques often fall short when tasked with identifying novel, unfamiliar correlations. In this context, we introduce Representation Analysis a global approach for interpreting a model's decision-making process. This approach is based on the analysis of the internal reprosentations within the models, as well as their interrelationships. After selecting a particular layer, relationships between representations are measured using the proposed EA distance measure. This enables the visualization of the representation space, the analysis of groups of neurons that have learned similar concepts, and the identification of representations that encode anomalous abstractions.
Depending on the problem specifics, the F.A distance computation may be executed in data-aware mode using the EA, distance metric or in a data-agnostic mode using the EA, distance metric. We denote the latter scenario as DORA Data-agmOstic Representation Analysis. The choice between a data-aware and a data-agnostic scenario primarily hinges on the availability of data for analysis. DORA reduces the dependence on data, proving particularly advantagoous for interpreting models for which the training data is either unavailable exceedingly difficult to obtain. Conversely, in practical applications, n-AMS are generally easier to comprehend compared to A-MMS.
Figure 5 provides an overview of the three primary stages of the Representation Analysis pipeline, each of which is further elaborated in the subsequent sections. Initially, layer of interest is selected from the Extreme-Activatioo model. Subsequently, the Extreme- Activation distances between representations are computed. Finally, the relationships between the representations are analyzed. This includes visualizing the representation space via Representation Atlases, automatically identifying outlier representations, and, if necessary, conducting a thorough manual examination of the causes behind the relationships between representations. As we will show in the following experiments, these outlier representations often encode undesired concepts.
Visualizing Representation Spaces with Representation Atlases
Inspired by Carter et al. (2019), the visual examination of the functional diversity within one layer can be donc by employing the dimensionality reduction method based on the pre-computed EA distance matrix. Such a visualization, referred as represenlation allas, allows researchers to visually examine the topological
11

---

Published in Transactions on Machine Learning Research (06/2023)
landscape of learned representations and identify clusters of semanti- cally similar representations. In the scope of this paper, we employed the widely used UMAP dimensionality reduction algorithm Mclnnes ct al. (2018). which has ostablished itself in recent years as an effective method for visualizing relat ionships between data points. Figure depicts the representation atlas of the output logit layers of the ResNet18 model trained on ImageNet. Each point in the figure corresponds to individual neural representation among the 1000 representations in the output layer. The color of each point reflects the WordNet hypernym, high-level synset, that corresponds to the learned concept of the particular representation. The UMAP visu- alization, based on the computed EA, distances, reveals the clusters of semantically similar representations that are proserved, which can be observed in Figure G
-.a Other
14
Bird
Mammaal
30
Dog
Artinet
UMAP3
Figure 6: Representation Atlas of the ResNet18 Logit Layer The fig- ure presents a UMAP visualization of the layer-wise EA, distances between output logit representations from a ResNet18 trained on ImageNet. Each point represents a unique class repre- sentation and is color-coded according to its corresponding WordNet hyper- nym, i..., the broader category.
In comparison with other dimensionality reduction methods, such as t-SNE Van der Maaten and Hinton (2008) and PCA Jolliffe and Cadima 2016). UMAP is scalable, exhibits faster computation time McInnes et al. (2018); Trozzi et al. (2021 Becht. et al. (2019;
Wu et al. (2019), and has fewer parameters to tune. Qualitatively,
compared to the other methods, UMAP was reported to improve
visualizations and accurately represent the data structure on the
projected components Trozzi al. 2021
Becht ct al. (0099) Wu
et al. 2019).
Identifying Outlier Representations
Despite their proven effectiveness across var- ious applications, Deep Neural Networks re- main susceptible to learning unintended arti- facts and undesired concepts from data. One potential application of our proposed dis- tance measure is the identification of anoma- lous representations, which deviate from the majority of representations within layer. This can be achieved by applying Outlier Detection methods Ruff et al. (2021) based on the computed distance matrix. We hy- pothosize that such anomalies include rep- resentations that have learned spurious con- cepts from the data. In subsequent experi- ments, demonstrate the efficacy of the FA distance measure in detecting such outlier representations ions under controlled conditions. During our practical experiments, we found that while some outlier representations learn unique, tesk-relevant concepts, these diver- gent representations often encode undesirable concepts, demonstrating shortcut learning or
-- 6.95 5 8.92
3.89, 7.13
mokel Aocpommana .0
Ressee "brcommn.1
2
8.77,1
9.79 4.11
mAMs "sae.. 0L"
10
1. ResNet 18, "safe. 1.01'
Figure 7: Functional similarity between representations due to shared watermark dependencies. Two represen- tations from the ResNet18 model, specifically "Broom" and "Safe", show significant co-activation on some n-AMS, despite their visual dissimilarity, which would typically suggest orthog- onal RAVs. A closer inspection of the mutual n-AMS reveals that the common feature the presence of watermarks on the images, which is shown among 4 n-AMS, mutually activating both representations.
Clever Hans behavior.
Investigating Individual Relationships
In order to determine the cause behind specific EA distance, individual relationships between representations can be examined manually. This can be accomplished through visualizing RAVs and AMS, as discussed
12

---

Published in Transactions on Machine Learning Research (06/2023)
in Section 3.3 For instance, Figure illustrates the spurious correlation between the "Broom" and "Safe" representations from the output logit layer of 18 which accounts for the low EA distance between these representations. Upon examining the shared n-AMS, it becomes clear that the n-AMS, which maximally activate both representations, contain a single common feature watermarks. Such visualizations can aid in investigating the underlying functional similarity between representations and offer insight into the reasons behind such similarities.
5
Evaluation
The practical utility of a distance metric between representations fundamentally depends on its capacity to gauge the similarity between concepts that various model representations have Iearned. The manner in which this similarity is assessed can lake on multiple forms, but from a standpoint of explainability, it is important that the distance metric is aligned Muttenthaler et al. (2022): : Gabriel (2020) with human perception the computed distances should resonate with our human senses of similarity and difference. Assuming that effective distance metric should label
we know what abstractions two representations are detecting, representations that detect concepts perceived as similar by humans as alike, and those identifying concepts perceived as distinct by humans as dissimilar. Such attributes of the distance metric facilitate the clustering of representations that are conceptually similar and the identification of outlier representations that encode concepts anomalous to the task hand. This has advantages as it aids in revealing shortcuts and Clever Hans effects, which often appear as out-of-distribution anomalics and are unnatural to the assigned task for example, textual watermarks in the context of object classification Lapuschkin et al. (2019) or specifie tokens in medical image classification Geirhos et al. 2020).
To quantitatively evaluate the alignment, we compared human-defined semantic distances between concepts, which we refer to as semantic baselines, with distance matrices computed bet ween representations trained to learn these concepts. For our study, we utilized two prevalent computer vision datasets, namely ILSVRC-2011 Russakovsky et al. (2015) and CIFAR-100 Krizhevsky (2009). : We established mcasure of distance between concepts by utilizing semantic distances between the class labels. This was accomplished by mapping the classification labels to entities within the WordNet. taxonomy database Miller 1995), a lexical database that organizes English words into taxonomy of synonym sets, synsets. In this taxonomy, each synset represents group of words that are synonyms or have the same meaning. WordNet organizes these synsets into a hierarchy, with more specific concepts being nested under more general ones.
For the ImageNet dataset, class labels were automatically mapped to the corresponding WordNet synsets due their inherent linkage, whereas for the CIFAR-100 dataset, the labels were manually matched to their respective synsets. is important to note that semantic distance does not directly equate to the visual similarity between concepts. However, positive correlations between semantic and visual similarities have been reported, thereby demonstrating significant positive relationship between semantic and visual distances Deselaers and Ferrari 20011 Brust and Denzler (2019).
Given the WordNet taxonomy in a form of a graph g = (V, E) with root r E V, the bascline semantic distances between entities from the WordNet database were computed using the following three distance measures:
Shortest-Path distance
Given two vertices C E V the distance between vertices is determined by the length of the shortest. path that connects the two entities in the taxonomy
dsp(c, c;) = I(c,,,
where I(s, 0z) is the function that returns the minimal number of edges that need to be traversed to get from oj cj.
distance Leacock and Chodorow (1998
13

---

Published in Transactions on Machine Learning Research (06/2023)
Given two vertices Cj V the distance between vertices is determined by logarithm of the shortest-path distance with additional scaling by the taxonomy depth:
dLo(C, = log ((e,, +1
1
log
2T
27
where T max,ev I(r, c) is the taxonomy depth.
Wu-Palmer distance Wu and Palmer 1994)
Given two vertices c; Cj E V the Wu-Palmer distance is defined as:
1(r, 1(r,ee((c,,
dsr(e, =1 -1- - ((r,,))((r,())
where Ics(ci, (g) is the Least Common Subsumer Pedersen et al.
2004) of two concepts and cj-
Furthermore, we have utilized the textual labels from both ImageNet and CIFAR100 datasets and calculated the Word2Vec Mikolov et al. (2013) similarity between class labels.
Werd2Vec distance
Given textual labels 4, t of two concepts Cj, we define Word2Vec distance as
1 V 44,,))
dw2v
/2
where cosw2v (A, B) is the cosine of the angle between Word2Vec embeddings of the words A, B.
EAg layer-wise, ResNet18
Shortest-path distance
Leacock-Chodorow distance
Wu-Palmer distance
Word2Vec distance
20 36
150
20
Figure 8: Comparison between EA, distance matrix and Seamntic Baselines: From left to right: the FA, distance metric computed for the output logits of the ImageNet pre-trained ResNet18 model, Wu-Palmer distances from WordNet taxonomy, and Word2Vec distance.
Shortest-Path,
Figure illustrates the comparison between the functional EA, distance matrix, derived from 1000 represen- tations from an TmageNet-trained ResNet network, and semantic baselines these are the human-defined distances between ImageNet concepts. To evaluate the alignment between the proposed distance metric and human-defined baselines, we employed the Mantel Test Mantel (1967). which is often applied in ecology and evolutionary biology to measure the correlation between two distance matrices. The test calculates the correlation cocfficient P: which indicates the strength of the relationship between the two matricos, and the p-value of the (est, which describes the statistical significance of the correlation.
It is essential to note that while we evaluate the alignment based human-defined semantic benchmarks, optimizing such metrics should not be the ultimate objective when proposing new distance metrics between representations. This is because DNNs can naturally employ different decision-making strategies than humans, and these differences may not always be attributed to spurious correlations. For instance, taxonony-been based approaches might be sub-optimal compared with attributing freedom to the models train for the desired tasks Binder al. (2012). Conversely, in Computer Vision, network representations are expected to be aligned to some extent due to the correlations between the visual and semantic similarity of classes.
14

---

Published in Transactions on Machine Learning Research (06/2023)
5.1 Hyperparameter selection
This section examines the selection of parameters in terms of their ability to attain optimal alignment with the semantic basclines. In empirical analysis, utilized pre-trained RosNet18 model on ImageNet, along with the ILSVRC-2012 validation set consisting of 50,000 images and 1,000 classes, employed for the data-aware metries. Herein, calculated the distance metrics between output logit representations, that is, the pre-softmax representations. For data-aware metrics, outputs of representations underwent normalization as discussed Section 3 Conversely, for the data-agnostic distance metric, specifically EAg, no normalization procedure was undertaken.
Minkowski distance
To investigate how different values of the parameter p affect the coherence to semantic baselines, varied the parameter and evaluated the alignment with four semantic baselines for each case. Figure 9 shows the effect of parameter selection on the Mantel test statistic. We observed that the optimal average value of the statistic across the four baselines was achieved for p = 2. However, for future experiments, we selected the second-best parameter choice with p due the natural connection between Euclidean distance and Pearson correlation. We also observed that higher values of generally result in lower alignment, possibly due to scnsitivity to the large amplitudes of individual datapoints.
Minkowski metric aligment, ResNet18
Mantel test statiscic
6.5
64 63
Shorassk-has
Washulmer
6.0
WareWWee
parameter
Figure 9: Impact of Parameter Selec- tion in Minkowski Distance on Align- ment with Semantic Baselines. To assess the aligmment with respect to the four semantic basclines, we calculated the Minkowski distance on the output logits of the ResNet18 network while varying the pa- rameter p. The Mantel correlation statistic is reported for each semantic baseline each parameter value.
EA, distance
EA,, is influenced by two key parameters: n, which denotes the number of n-AMS signals gathered, and d, which represents the size of the subset collected from each signal. To investigate the impact of parameter selection, we varied these parameters for both pair-wisc and layer-wise modes. Figure I0 shows the average Mantel correlation statistic across four semantic baselines for each hyperparameter choice. Our observations reveal that, in general, incrcasing the number of collected n- AMS, irrespective of the parameter d, has positive impact on the alignment. However, the optimal depth d is achieved when n-AMS are taken from subsets of d = 50 datapoints.
EA, distance
In the data-agnostic version of the Extreme-Activation distance, the choice of hyperparameters depends on the s-AMS generation method used. In our study, we employed the Feature Visualisation method to generate AMS, and we identified two critical hyperparameters: n, which the number of generated s-MMS per representation, and m, which is the number of optimization epochs per signal. Figure [T] depicts the impact of the EA, distance measure's hyperparameter selection on semantic bascline alignment. We observod that while increasing the number of generated s-AMS generally has positive effect, this effect negligible compared to the positive impact of increasing the number of optinization epochs per representation. This is likely due to the generation algorithms' convergence to better local optima, resulting in improved visual preciseness of the images, as illustrated on the right side in Figure
5.2 EA, Preserves the Angles Between Natural Representation Activation Vectors
Although both n-AMS and AMS activate specific neural representations maximally, the adversarial nature of synthetic signals needs to be considered. In our experiments, observed that while the generated s-AMS
15

---

Published in Transactions on Machine Learning Research (06/2023)
E pair-wise
EAn layer-wise
0
0.1
0.09
0:32
0.34
0.35
0.35
0.35
0.35
0.35
0.01
0.1
0.11
0.3
0.34
0.34
0.37
0.37
0.38
0.38
n, nimbr of 1-AM5
5
0
0.24
0.33
0.44
0.44
0.43
0.43
0.42
0.39
0.39
n. n.mer of n-AMS
5
-0.01
0.24
0.33
0.48
0.48
0.48
0.48
0.48
0.45
0.45
10
0.32
0.42
0:46
0.48
0.45
0.44
0.43
0.4
19
0.33
0:45
0.51
0.51
0.5
0.5
0.49
0.47
50
0.45
0.48
0.49
0.48
0.47
0.45
0.44
50
0.49
0.52
0.54
0.54
0.53
0.52
0.51
100
0.47
0.48
100
0.5
0.53
0.55
0.54
0.53
0.52
200
200
0.01
0.51
0.54
0.55
00
10
50
100
200
500
1000
5000
10000
50
100
200
500
1000
5000
10000
d. n-MMS depth
d. n-AMS depth
Figure 10: Impact of parameter selection in EA, distance on alignment with semantic baselines. To assess alignment, with four semantic baselines, we calculated data-aware FAA distance on the output logits of the ResNet18 network while varying the parameters n and d for both pair-wise and layer-wise options. The average Mantel corrclation statistic across four semantic basclines is reported each cell.
EAg pair-wise
EAg layer-wise
1.0
0.45
0.8
'herr"
0.087
0.13
0.25
0.36
3.37
0.37
0.36
0.14
0.22
0.26
0.42
0.45
0.46
number of 5ANS
0.065
0.15
0.26
0.37
3.37
0.30
0.38
0.24
0.3
0.04
5.06
0.47
0.67
0.37
0.30
0.67
0.4
980, 'volcano"
0.077
0.27
0:37
0.30
0.30
0.18
0.25
0.31
0.64
3.46
0.47
0.063
0.16
0.26
0.37
3.98
0.38
0.38
8
0:18
0.25
0:31
0:44
9.46
0.47
octimisation
50
500
number optim optimisation for
10
250
500
-0.0
10
Number
500
Figure 11: Impact of the parameter selection EA, distance on alignment with semantic baselines. evaluate the alignment between the EA, distance and four semantic baselines, we computed the data-agnostic EA, distance using the ResNet18 network's output logits while varying the hyperparameters and for both pair-wise and layer-wise options. For each cell, we reported the average Mantel correlation statistic across the four semantic baselines. The effect of the hyperparameter m, which corresponds to the number of optimization steps taken for s-AMS generation, on two neural representations from the ResNet18 output logit layer is shown on the right.
are far from the original natural image domain, the angles between natural and synthetic RAVs are consistent, providing additional evidence to the utility of the EA, distance metric.
To evaluate the angle conservation quantitatively, dataset and computed EA, and EA, distances between the output logit representations, ie., all 1000 ImageNet classes. It's important to mention that the EA, distance is calculated over normalized reprosentations, while EA, is based on the unnormalized output of representations. In this experiment, we kepl the number of signals constant at n = 50 for both distance metrics, yet varied the parameter d for n-AMS generation and the parameter m for s-AMS generation. Given F (fi....,fe fx), which corresponds the ResNet18 output layer with k 1000 neural representations, we evaluated the Root Mean Square Error (RMSE) between pairwise EA,, and FA, distances:
employed ResNet18 pre-trained the ImageNet
16

---

Published in Transactions on Machine Learning Research (06/2023)
m, number of optimization stepsfor
0.58
0.71
0.71
0.71
0.71
0.71
0.71
140
'Samoyyd's-AMe
5
0.36
0.19
0.14
0.13
0.13
0.13
0.13
ReaNet 18, 18.'Samoyed
120
+ m 200
10
0.36
0.19
0.14
0.12
0.12
0.12
0.12
100
+ m 500
"Husky's-AA s-AMM
50
s-AMS
200 100 50
0.36
0.17
0.11
0.088
0.088
0.089
0.089
80
+ 200
530
60
0.35
0.17
0.1
0.074
0.074
0.075
0.075
40
0.35
0.16
0.099
0.064
0.064
0.064
0.064
yx
500
0.35
0.16
0.093
0.053
0.051 0.051
0.052
20
xy
10
50
100
200
500
20
40
60
80
100
120
d, depth of n-AMS
ResNet 18, Siberian husky'
Figure 12: Similarity and angle preservation between EA,, and EA, distance measures. The left part of the figure shows the RMSE (lower is better) between pair-wise EA, and EA, distances on the output layer of the ResNet18 network, with a fixed parameter n = 50 for both metrics, while varying parameters d, corresponding to the subset size in n-AMS sampling, and m, number of epochs for s-AMS generation. On the right part of the figure, the distributions of pair-wise activations of A-M signals are visualized with different paramcters m for two neural reprosentations, namely "Samoyed" and "Siberian husky", overlayed with the direction of natural RAVs computed with 50 and 1000. The length of natural RAVs was
extended to enhance visibility.
Σ
k Ei=it1 (f. EEA (fi, f2)2
RMSE
(13)
k(k 1)/2
where k(k- - 1)/2 corresponds to the number of all unique pairs of two different functions from set of k
functions.
Figure 12 illustrates the similarity between the computed EA, and EA, distances between the representations of the 1000 ImageNet classes. In the left part of the figure, which shows RMSE between the two distance measures for different parameters, observe that for cach parameter m for EA, distance, the lowest error is achieved with an EA, distance with high values of d. This suggests that the EA, distance accurately captures the angle between RAVs corresponding to the images with the highest activation percentile. Additionally, we observed that increasing the parameter m is beneficial to lowering the RMSE between natural and synthetic measures. Furthermore, the right part the figure shows the direction of natural RAVs and activations of s AMS for "Samoyed" and "Siberian husky" representations from ResNet18. From this figure, we can observe that the angle between natural RAVs and synthetic RAVs conserved.
5.3 Evaluating the alignment with human judgment
In this experiment, we quantitatively assess the alignment of the discussed distance metrics with the human- defined distance measures across different datasets and architectures. To this end, we employed eight different architectures for two datasets, TmageNNet and CTFA R100. For ImageNet, we employed ResNet18 HH al. (2016), AlexNet Krizhovsky et al. (2017), VIT Dosovitskiy et al. (2020), BEiT Bao et al. (2021). Inception V3 Szegedy et al. (2016), DenseNet. 161 HHann et al. (2017), MobileNet V2 Sandler et al. (2018). ShuffleNet. V2
Ma et al. (2018), while for CIFAR-100 100, we used ResNet 18, ResNet 9, MobileNet V2, ShuffleNet V1, and
V2, as well as NASNet Qin and Wang (2019). SqueeeNet Tandola et al. (2016) and VGG 11 Simonyan and
Zisserman 2014).
We computed functional distances with optimal hyperparameters found Section 5.1 including Minkowski p = 1, Pearson, Spcarman, EA, with n 50, 200, and EA, with n = 3, m 500, on the output logit layer for each model. We then compared each distance matrix with four semantic baselines: Shortest-Path, Leacock-Chodorow, Wu-Palmer distances from WordNet taxonomy, and Word2Vec distance. This comparison
17

---

Published in Transactions on Machine Learning Research (06/2023)
Table Alignment of Distance Metrics in ImageNet Trained Models: Each cell represents the average Mantel test statistic (higher is better) across four semantic basclines: Shortest-Pathe Leacock- Leacock-Chodorory Wu-Palmer distances, and Word2Vec distance. All results demonstrate statistical significance with 0.001.
Minkowski
Pearson
Speorman
E.n
EAg
p
p-w
(-w
P-W
b-w
ResNet18
0.49
0.50
0.48
0.49
0.55
0.38
0.47
BelT
0.32
0.36
0.29
0.44
0.50
0.39
0.47
Mobilenet
0.46
0.46
0.45
0.47
0.52
0.40
0.50
DenseNet161
0.46
0.47
0.44
0.49
0.54
0.32
0.39
SlufDeNeeVV
0.21
0.21
0.19
0.29
0.30
0.19
0.16
Inception V3
0.31
0.34
0.32
0.38
0.49
0.22
0.27
AlexNet
0.52
0.53
0.52
0.52
0.55
0.42
0.45
ViT
0.53
0.54
0.52
0.54
0.58
0.48
0.53
Mean
0.41
0.43
0.40
0.45
0.50
0.35
0.40
Table 2: Alignment of Distance Metrics in CIFAR100 Trained Models: Each cell represents the average Mantel test statistic (higher is better) across four semantic baselines: Shortest-Path, Leacock- Chodorow, Wu-Palmer distances, and Word2Vec distance. All results demonstrate statistical significance with p < 0.001.
Minkowski
Pearson
Spearman
EAn
EAg
p
p-w
I-w
P-W
b-w
ResNet9
0.32
0.37
0.33
0.41
0.52
0.27
0.30
ShuffleNet
0.49
0.52
0.49
0.53
0.59
0.43
0.47
MobileNetV2
0.50
0.51
0.49
0.52
0.59
0.40
0.44
ResNet18
0.43
0.47
0.45
0.48
0.57
0.30
0.37
ShuffleNet
0.48
0.51
0.49
0.52
0.58
0.42
0.46
VGG11
0.30
0.31
0.31
0.36
0.43
0.23
0.23
NasNet
0.48
0.51
0.48
0.52
0.59
0.36
0.41
SqueezeNet
0.50
0.52
0.51
0.53
0.59
0.45
0.51
Mean
0.44
0.46
0.44
0.48
0.56
0.36
0.40
yielded four Mantel test statistics per distance metric. The results of the evaluation are presented in Table for ImageNet-trainee models and in Table 2 for CIFAR100 models, where averaged the four Mantel correlation lest statistics for each model and distance metric. Our analysis indicates that the layer-wie wise EA, metric's distance is generally more favorable due to its stronger linear relationship with all four baseline metrics. Furthermore, we observed that the data-agnostic EA, metric is on par with data-aware metrics in terms of alignment with the semantic baselines.
5.4 Evaluating Anomaly-ldentificatio capabilities
The alignment of distance metrics between neural representations and human judgment of concepts presents intriguing potential application. Specifically, anomalous compared to the majority of learned representations, based on the functional distance. While these representations may simply learn unique individual concepts, we demonstrate in further experiments that in real-life scenarios they might correspond to the undesired concepts from spurious correlations in the training data that diverge from the typical (intended) decision decision-making strategy.
identify representations that semantically
To assess the usefulness the aligument between distance metrics and human-defined semantic baseline, conducted the experiment, where measured the ability of the distance metrics to detect anomalous
18

---

Published in Transactions on Machine Learning Research (06/2023)
representations. For this purpose, we trained ResNet18 network combination of two conceptually different. datasets. The combined dataset comprised the Tiny Imagenet Le and Yang 20155) containing 200 ImageNet classes, and the MNIST numbers, resulting in a total of 210 classes. MNIST images were upsampled to the size of 3 x 64 x 64 pixels to match the size of images in Tiny ImageNet. After training on the combined dataset the image classification task, we computed functional distances between the output logits and evaluated the ability of different Outlier Detection (OD) methods to detect MNIST logits, given the computed distance matrices only. For this, we utilized five different Outlier Detection methods: the Angle-based Outlier Detector (ABOD)
cettwrittennnuubeer dataset Deng (2012), containing 10 handwritten
Kriegel et al. (2008), Feature Bagging (FB) Lazarevic and Kumar (2005), Tsolation Forest (IF) Liu et al. (2008), Local Outlier Factor (LOF) [reunig et al. (2000) and One-class SVM (OCSVM) Schölkopf et al. (2001) We evaluated the performance of the Outlier Delection methods using the AUC ROC metric for the binary classification between Tiny ImageNet and MNIST representations. To ensure stability in light of the stochastic nature of some outlier detection methods, the results of the outlier detection were repeated 100 times with different random stales
EA, pair-wise
EA, pair-wise
TinylmageNet S-AMS
MNIST S-AMS
0.8
0.6
0.4
o
0.0
Figure 13: Chomaly-Detectinn Evaluation Experiment. From left to right, pair-wise F.A,, distance matrix between the output logits of the network trained on the combined dataset, the EA, distance matrix, s-AMS for Tiny ImageNet logits, and s-AMS for MNIST logits. MNIST representations both distance matrices in the bottom right corner, revealing a block structure in both distance metrics that suggests high degree of functional differences between Tiny ImageNet representations and semantically distinct MNIST representations. On the left, we can visually observe differences between the 8- &-AMS of Tiny ImageNet and MNIST representations.
highlighted on
We utilized the same hyperparameter configuration for distance computation described in Section 5.3 The effectiveness of EA distances, both natural and synthetic, in distinguishing between representations of Tiny ImageNet and MNIST is demonstrated in Figure 13 as evidenced by the block structure of the distance matrices. This behavior can be attributed to the visual dissimilarities between the classes, where Tiny ImageNet classes exhibit natural and diverse features that are typical for natural images, while MNIST images consist of white digits on black background. Tn the case of synthetic EA distance, the ability detect MNIST representations based on the visual differences in the s-AMS, which are depicted in the right-hand portion of Figure [3 The M-AMS-aased EA distance measure depends on the network's ability to perceive self- reff-genertted s-AMS, and Tiny ImageNet classes, which contain high-level natural concepts, and the more data-specific patterns for MNIST classes, which illustrate the network's perception of white-on-black handwritten digits and letters.
observe distinct dissimilarities between the patterns of s-MMS for
The results of the described experiment are presented in Table 3 which indicate that, in general, all distance metrics are capable of detecting MNIST representations. However, the EA distance metrics are generally more effective in detecting semantically artifactual representations, whereas the pairwise EA,, metric is the most effective.
19

---

Published in Transactions on Machine Learning Research (06/2023)
Table 3: Anomaly Identification Performance of Distance Metrics. The table displays the average AUC ROC (higher is better) binary classification performance of the Outlier Detection methods across 100 re-trials, in the task of detecting MNIST representations among the combined Tiny ImageNet and MNIST representations, specifically in the output layer of the trained network.
Minkowski
Pearson
Spearman
EAn
EA.
p-w
1-w0
p-w
1-
ABOD
0.56
0.63
0.58
0.91
1.00
0.82
0.71
FB
0.97
0.99
0.81
1.00
1.00
0.89
0.87
IF
0.83
0.87
0.64
0.94
0.70
0.76
0.61
LOF
0.65
0.53
0.55
0.67
0.96
1.00
0.87
OCSVM
1.00
1.00
0.95
1.00
0.67
1.00
0.72
Mean
0.80
0.80
0.71
0.90
0.87
0.89
0.76
6
Experiments: Finding Outlier Representations with DORA
In this section, we illustrate the broad applicability of the DORA framework and demonstrate that outlier representations, often found in intermediate layers, can frequently encode malicious and undesirable concepts.
6.1 ImageNet pre-trained networks
Pre-trained networks on ImageNet have become an essential component in the field of Computer Vision. Their capability to recognize a diverse set objects and scenes makes them particularly useful as a starting point for a wide range of computer vision tasks. They are frequently utilized for fine-tuning specific tasks as feature extractor, where the images are encoded by the networks for further computations Zhuang et al. 20200; Weiss et al. (2016).
In the following, we explore the feature extractor representations of three widely-used pre-trained models: ResNet18 He et al. (2016), MobileNetV2 Sandler et al. 2018)) and DenseNet121 Huang et al. (017)). Using LOF outlier detection, we found latent layers with representations that appear to be walermark detectors, e-g., detecting Chinese and Latin text patterns, As ImageNet does not have a specific category for watermarks, these representations could be scen as Clever-Hans artifacts and deviate from desired dccision-making Lapuschkin et al. (2019); Anders el al. (2022). To verify these representations can detect watermarks, created two binary classification datasets, for Chinese and Latin watermarks, containing normal images and identical images, with inserted random watermarks, evaluating the sensitivity of individual representations using the AUC ROC classification measure. To ensure the detection of characters and not specific words /ats//hhssssssse (unlike CLIP models Goh et al. (202T)), the probing datasets were generated with random characters (for more details we refer to the Appendix). Our results show that not only the reported outliers but also neighboring representations in EA distance are affected by artifactual behavior. Lastly, we find that this behavior persists during transfer learning, posing a risk for safety-critical fields like medicine.
ImageNet ResNet18
We applied DORA to analyze the Average Pooling layer of the ResNet18 model, which consists of the k 512 high-level representations that are commonly used without further modification during transfer learning. Following the DORA approach, each representation and with 500, based on our findings in the section 5.1 After calculating the EA, distances, we used the LOF method with a contamination parameter p 0.01 (corresponding to the top 1% of representations), and the number of neighbors was set to 20.
calculated EA, layer-wise distance with s-AMS per
LOF identified five outlier representations, namely neurons 7, 99, 154, 160, 162, and 393. The outlier neuron 154, displayed specific, recognizable pattern in s-AMS that could be perceived as the presence of Chinese logograns. By probing the network on binary classification problem between images watermarkod with Chinese logograms normal images, Neuron 154 showed the highest detection rate (AUC ROC of
20

---

Published in Transactions on Machine Learning Research (06/2023)
154
129
Receiver Operating Characteristic
1.0
347
0.6
0.4
P251, AJC 0.80
AJC. 0.94
129 154
282
0.2
2438,
S1. AUC
9.76
347
459
61
439
282
2.87,
439
81
10
00
0.6
0.6
100
Faise Positive Rate
Figure 14: Cluster of Clever-Hans representations in the ResNet18 feature extractor. From left. to right: representation atlas of the ResNet18 average pooling layer with the highlighted cluster of Clever-Hane represcntations (left), s-MMS of the represcntations in the cluster (middle), and AUC ROC scnsitivity scores for the delection of images with Chinese watermarks in the binary classification problem (right), where colored curves correspond to the behavior of representations in the cluster and gray curves for other representations. From the s- s-AMS of neuron 154, we can observe symbolic patterns resembling Chinese logograms learned by the neuron as well as by its closest neighbor neurons. We can observe that the outlier neuron 154 exhibits the highest AUC value (green curve), followed by its nearest neighbors.
0.94) towards the class with watermarked images, providing significant evidence that this reprosentation is susceptible to the Clever-Hans effect. Further analysis of neighboring representations in EA, distance showed that they also exhibit similar behavior. The results of the analysis of the ResNet 18 average pooling layer are shown in Figure [4 illustrating the cluster of Clever-Hang representations found, along with their s-AMS the binary classification problem. Additional information on the dataset
and AUC ROC performance generation and the identified outlier representations can be found in the Appendix. Note that in general, the presence of such artifacts could indeed pose scrious risks and may lead to degradation in classifier performance (see Anders et al. (2022)
In the further investigation of the model, we inferenced s- A-M signals of representations in the reported CH-cluster and obtainod their predictions by the model. Among the selected signals, the model prodominantly predicted an affiliation of these signals with the classes "carton", "swab", "apron", "monitor" and "broom", which is in line with the reported spurious correlation of the "carton" class and Chinese watermarks Li et al. )022). Upon computing the corresponding s-AMS signals for these logits, we were able to confirm their association with CH-behaviour, as they displayed clear, visible logographic patterns, specific to Chinese character detectors, in their corresponding s-AMS. Corresponding signals and additional information could be found in Appendix.
ImageNet MobileNetV2
We used DORA with the same parameters as the previous experiment (n s-MS per each representation and m 500 epochs for s-AMS generation) to analyze the "features" layer of MobileNet network [Sandler et al. 2018 which consists of k 1280 channels with 7 activation maps. The analysis was performed on channels by averaging the resulting activation maps of neurons. We calculated the EA distances between representations and applicd the LOF method with a contamination parameter of 0.01 which yielded 13 outlier representations. Upon visual inspection of the S-MMS of these representations, we observed distinct patterns specific to Chinese character detectors in neurons 397, 484, 806, and 1131. Figure [5 illustrates the s-AMS of these neurons, as well as the sensitivity of neurons in the Chincso-charactorr detection task. We can observe that the neighbors of these neurons (397, 484, 806, 1131) are sensitive CH artifacts and form a distinctive cluster visible in the representation atlas.
21

---

Published in Transactions on Machine Learning Research (06/2023)
Reported LOF outliers (p 0.01. 20)
AUC ROC scores
S-AMS of outlier representations
8D6
484
AUC
D 67
A
UMMP
1131
397
-3
UMAP
UMAP
Figure 15: Cluster of Clever-Hans representations in the MobileNet V2 feature extractor. The left figure illustrates the outlier representations identified by the LOF OD method, overlaid the DORA representation atlas. The middle figure displays the sensitivity of the neural representations to Chinese watermarks, where the highly-scnsitive cluster of neurons can be clcarly observed in the bottom-right part of the atlas, including 3 reported outlier representations. The right graph illustrates the S-MMS of several of the reported outlier neurons, which exhibit a distinctive logographic pattern typical of Chinese character detectors.
227
733
Receiver Operating Characteristic
1.0
Podtee Rd0
0.
62
507, AUC
863 AUC 5.33
0.4
380PP1
Rube
Figure 16: DenseNet121 Latin text detector. Applying DORA to the last layer of the feature extractor of DenscNet121 yields, among others, Neuron 427 outlier, which corresponds to the upper left of the 4 feature visualizations. From neuron 427 well as from its three closest neighbors (shown left), we can observe semantic concepts resembling Latin text characters. The AUC values were computed using the average channel activations on the Latin probing dataset. As shown, the AUCs are high for the representation outliers found by DORA, compared to most of the other representations, which indicates that they indood learned detect. Latin texl patterns.
ImageNet DenseNet 121
We conducted similar analysis on the last layer of the feature extractor of the ImageNet pre-trained DenseNet121 model, which consists of k 1024 channel representations with 7 x 7 activation maps. We calculated n = 5 s-AMS per representation with m 150 optimization steps for faster experimentation. The LOF outlier detection method with contamination parameter of 0.01 identified 10 outlier representations. One of these, neuron 768, was found to be Chinese character detector (more information can be found in the Appendix). By increasing the contamination paramoter to p p= 0.035 (corresponding the top 3.5% or representations), we also identified neuron 427, which susceptible to the detection of Latin text and watermarks. Figure 16 illustrates the representation atlas, highlighting representation 427 along with several
22

---

Published in Transactions on Machine Learning Research (06/2023)
neighboring representations, namely neurons 733, 507, and 463, which also exhibit high detection rate for unintended concepts.
Given the widespread use of pre-trained models in safety-critical areas, it is essential that the artifacts embodied in a pre-trained model are made ineffective or unlearned during the transfer learning task (see also Anders et al. (2022)) ). To this end, we examined the effect of fine-tuning the pre-trained 1enseNet121 model on the CheXpert challenge Irvin et al. (2019), which benchmarks classifiers on a multi-label chest radiograph dataset. Despite the modification of all model parameters during fine-tuning, neurons 427 and 768, which were Latin and Chinese characters detectors in the pre-trained model, retained their original artifact-detection capabilities and remained outliers after applying DORA. We studied neuron 427's ability to detect Latin text and found that it had an AUC value of 0.84 in the pre-trained model and 0.81 in the fine-tuned model. Similar behavior was observed with neuron 768, indicating that the Clever- Hans effect persisted after fine-tuning.
6.2 CLIP ResNet50
CLIP (Contrastive Language- Language-Image Pre-training) models, which are designed to predict the associations between text and images, are trained using a contrastive learning objective Dai and Lin (2017); Hjelm et al. (2018) on extensive datasets. They are often fine-tuned for lasks like image classification Agarwal et. al. (2021) text-to-imag synthesis, where CLIP models frequently function text encoders (e.g., Stable Diffusion Rombach et al.
CLIP ResNet50, Layer 4, neuron 1865
SSMM
(2022)) )
In this experiment, we explored the representation space of the CLIP ResNet50 model Radford et al. (2021), with partic- ular emphasis on the final layer of its image feature extractor (referred to "layer 4"). While the training dataset was not
AM Images
publicly revealed, it is known to be significantly larger than standard computer vision dalasets like ImageNet, leading to broader range of concepts compared to ImageNet networks. We applied DORA to the 2048 channel representations from "layer 4', generating n 3 signals per representation with m = 512 and employing settings akin to those in (Goh et al. 2021
Figure 17: AMS for reported outlier representation. LOF identified neuron 1865 as the strongest outlier. Analysis of s- AMS and most activating images from Ima- geNet (obtained from OpenAI Microscope) indicate that it primarily detects white im- agos/backgrounds, which is atypical com- pared to other high-level representations.
Analysis of the outlier representations with contamination pa- rameter p 0.0025 yielded 6 outlier neurons, namely 631, 658, 838, 1666, 1865, and 1896. Representation 1865 neuron with the highest outlier score was found to detect the unusual as shown by s-AMS and
concept of white most activating images (collected from OpenAI Microscope) in the Figure 17. However, the other outlier representations could not be concluded to be undesirable as they seemed to detect rare but natural concepts. Further details and analysis of the other outlier representations can be found in the Appendix.
After computing the representation atlas for "layer 4", we manually investigated several distinctive clusters. Figure 18 illustrates the representation atlas alongside several reported clusters of semantically similar representations. With our analysis, we found a cluster of we were able to confirm the presence of geographical nourons, as reported in (Goh et al. (021] and we noted that representations from neighboring geographical regions, such as India, China, and Japan, were located close to one another. Additional information and more detailed visualizations can be found in the Appendix.
representations. Furthermore,
23

---

Published in Transactions on Machine Learning Research (06/2023)
1731
1376
1502
996
2011
255
ExplicitPornograaphiall representations
Money/Finance
1301
1340
1037
1898
Reptiles
UMAP]
Asia-geographical representations
CLIP ResNet50, Layer 2048 neurons
Figure 18: Representation atlas of CLIP ResNet50 "layer 4 Representation atlas for CLIP ResNet50 "layer 4", where several clusters of representations are highlighted. Activation-Maximizatiia signals associated with the Explicit/Pornograpiic representations were omitted due to the presence of explicit concepts in the signals.
7 Discussion and Conclusion
The popularity of Deep Neural Networks (DNNs) across diverse fields has brought to light the significant challenge posed by their inherent opacity, particularly for the fair and responsible deployment of DNNs. The presence of artifacts, spurious correlations, or biases in datasets not a rare occurrence. Therefore, it has become increasingly crucial to audit these models using Explainable Artificial Intelligence (XAI) methods to avert potential undesirable or even harmful behavior. To date, audits have primarily employed local XAI methods, which necessitate data access to elucidate the predictions of given model and are often found be limited when it comes to uncovering far as our kuowledge extends, there existed no methods for identilying representations that had inadvertently or malevolently learned unintended concepts.
potential biases Adebayo et al. 2002 Prior to research,
In our work, we introduced a general Representation Analysis pipeline for exploration of the relationships between neural representations within a specific layer and introduced DORA as a special case data-aggostic analysis. The core of our framework the newly introduced Aatreme-Acttattoo (FA) distance measure, which allows
measure the similarity between concepts, learned by representations within the model. This distance measure is casy to interpret and it allows to analyze relationships between representations in the layer, including the identification of the anomalous representations, that we demonstrate in practice often correspond to the undesired spurious concepts.
Although we have demonstrated the broad applicability of DORA, there exist several limitations that require attention. Firstly, the proposed approach assumes that undesired behavior in representations is not systematic. Consequently, DORA may not be able to identify infected representations if such behavior is widespread across a large number of representatious, as it would no longer be considered anomalous. Another limitation pertains to the potential semantic multimodality of representations Goh et al. (021 and additional research is necessary to address this issue.
In summary, DORA expands the scope for auditing "black-box' systems, thereby offering a methodology that enhances the understanding learned representations within the model and their interrelationships. By facilitating deeper dive into these complex systems, Representation Analysis makes it possible demystify the intricacies of the internal model behavior and consequently leads to more transparent and accountable machine learning process, encouraging robustness and trustworthiness in the deployment of these models.
24

---

Published in Transactions on Machine Learning Research (06/2023)
Acknowledgements
We would like to thank Filip Rejmus for his analysis regarding the visualization of the Clever Hans (CH) behavior in representations with global explanation methods. Furthermore, we would like to thank Sebastian Lapuschkin for fruitful discussions about CH behavior in Deep Neural Networks.
Funding
This work partly funded by the German Ministry for Education and Research through the project Explaining 4.0 (ref. 0115200551), the German Research Foundation (ref. DFG KI-FOR 5363), the Investi- tionsbank Berlin through BerDiBa (grant no. 10174498), and the European Unions Horizon 2020 programme through iTooBoS (grant no. 965221). KRM was partly funded by the German Ministry for Education and Research (under refs 011S14013A-E, 01GQ1115, 01GQ0850, 01IS18056A, 01IS18025A, and 01IS18037A) and BBDC/BZMT. and BIFOLD and by the Institute of Information & Communications Technology Planning Evaluation (IITP) grants funded by the Korea Government. (MSIT) (No. 2019-0-00079, Artificial Intelligence Graduate School Program, Korea University and No. 2022-0-00954, Development of Artificial Intelligence Technology for Personalized Plug-and-2 and-Play Explanation and Verification of Explanation).
References
GitHub
Practice on CIFAR100- github.com. pytorch-cifar100 2020. [Accessed OS-Jan-2023].
http:://gttaab..co//eeaaaaatt
J. Adebayo, M. Muelly, H. Abelson, and B. Kim. Post hoc explanations may be ineffective for detecting unknown spurious correlation. In International Conference on Learning Representations, 2022.
S. Agarwal, G. Krueger, J. Clark, A. Radford, J. W. Kim, and M. Brundage. Evaluating CLIP: towards characterization of broader capabilities and downstream implications. arXie preprint arXie:2108. 02818, 2021.
C. J. Anders, L. Weber, D. Neumann, W. Samek, K.-R. Müller, and S. Lapuschkin. Finding and removing clever hans: Using explanation methods to debug and improve deep models. Information Fusion, 77:261 295, 2022.
S. Bach, A. Binder, G. on, F. Klauschen, K.-R. Müller, and W. Samek. On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PloS one, 10(()x:0130140, 2015.
D. Bachrens, T. Schroeter, S. Harmeling, M. Kawanabe, K. Hansen, and K.-R. Müller. How to explain individual classification decisions. Journal of Machine Learning Research, 11(Jum):1803 1831, 2010.
H. Bao, L. Doug, and F. Wei. BEIT: BERT pre-training of image transformers. arXiv preprint orXiv:2106 08254, 2021.
D. Bau, B. Zhou, A. Khosla, A. Oliva, and A. Torralba. Network dissection: Quantifying interpretability of deep visual representations. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6541-65449 2017.
Ban, J.-Y. Zhu, H. Strobelt, B. Zhou, J. B. Tenenbaum, W. T. Freeman, and A. Torralba. GAN dissection: Visualizing and understanding generative adversarial networks. ar preprint arXiv:1811 10597, 2018
E. Becht, L. McInnes, J. Healy, C.-A. Dutertre, L. W. Kwok, L. G. Ng. F. Ginhoux, and E. W. Newell. Dimensionality reduction for visualizing single-cell data using umap. Nature biotechnology, 37(1):38 44, 2019.
Y. Bengio, A. Courville, and P. Vincent. Representation learning: A review and new perspectives. IEEE transactions pattern analysis and machine intelligence, 35(:))1798 18288, 2013.
F. BieBmaan, F. C. Meinecke, Gretton, A. Rauch, G. Rainer, N. K. Logothetis, and K.-R. Müller. Temporal kernel CCA and its application in multimodal neuronal data analysis. Machine Learning, 79(1):5 27, 2010.
A. Binder, K.R. Müller, and M. Kawanabe. On taxonomics for multi-class image calegorization. International Journal of Computer Vision, 99(3):281-38) 2010 2012.
25

---

Published in Transactions on Machine Learning Research (06/2023)
A. Binder, L. Weber, S. Lapuschkin, G. Montavon, K.-R. Müller, and W. Samek. Shortcomings of top-down randomization sanily checks for evaluations of deep neural network explanations. In Proceedings of the IEEE/CVE /VFF Conference on Computer Vision and Pallern Recognition, pages 16143 16152, 2023.
S. Bird, E. Klein, and E. Loper. Natural language processing with Python: anabyaing text with the natural language tollii. O'Reilly Media, Inc., 2009.
E. Bisong and E. Bisong. Google colaboratory. Building machine learning and deep learning models on google cloud platform: a comprehensive guide for beginners, pages 59 64, 2019.
R. Bommasani, D. A. Hudson, E. Adeli, R. Altman, S. Arora, S. von Arx, M. S. Bernstein, J. Bohg, A. Bosselut, E. Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.0788 2021.
J. Borowski, R. S. Zimmermann, J. Schepers, R. Geirhos, T. S. Wallis, M. Bethge, and W. Brendel. Natural images are more informative for interpreting cun activations than state-of-the-art synthetic feature visualizations. In NewrlPS 2020 Workshop SVRHM, 2020.
M. L. Braun, J. M. Buhmann, and K.-R. Müller. On relevant dimensions in kernel feature spaces. The Journal of Machine Learning Research, 9:1875-1900 1908, 2008.
M. M. Breunig, H.-P. Kriegel, R. T. Ng, and J. Sander. LOF: identifying density-based local outliers. In Proceedings of the 2000 ACM SIGMOD International Conference on Management of data, pages 93 03-104, 2000.
K. E. Brown and D. A. Talbert. Using explainable AI to measure feature contribution to uncertainty. In The International FLAIRS Conference Proceedings, volume 35, 2022.
T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advonces in Neural Information Processing Systems, 33:1877-1901, 2020.
C.-A. Brust and J. Denzler. Not just matter of semantics: The relationship between visual and semantic similarity. In German Conference on Pattern Recognition, pages 414-427. Springer, 2019.
V. Buhrmester, D. Münch, and M. Areus. Analysis of explainers of black box Deep Neural Networks for Computer Vision: A survey, arXiv preprint arXis:1911.11110 2019.
K. Bykov, M. M.-C Höhne, A. Creosteanu, K.-R. Müller,, F. Klauschen, S. Nakajima, and M. Kloft. Explaining Bayesian Neural Networks. arXiv preprint 0r0i:21008..003 2021.
K. Bykov, A. Hedström, Nakajima, and M. M.-C Höhne. NoiseGradennancing explanations by introducing stochasticity to model weights. In Proceedings of the AAAT Conference Artificial Intelligence, volume 36, pages 6132-61 0240, 2022.
N. Cammarata, G. Goh, S. Carter, L. Schubert, M. Petrov, and C. Olah. Curve detectors. Distill, 500):400,,,-003, 2020.
S. Carter, Z. Armstrong, L. Schubert, Johnson, and C. Olah. Exploring Neural Networks with activation atlases. Distill,, 2019.
C. Chen, O. Li, C. Tao, A. J. Barnett, J. Su, and C. Rudin. This looks like that: deep learning for interpretable image recognition. arXiv preprint orXiv:18066 1ott.100..11000 10574, 2018
J. Da. A corpus-based study of character and bigram frequencies in chinese e-texts and its implications for chinese language instruction. Proceedings of the fourth International Conference on new technologies in teaching and learning Chinese, pages 501-511. Citeseer, 2004.
B. Dai and D. Lin. Contrastive learning for image captioning. Advances in Neural Information Processing Systems, 30, 2017.
J. Deng, W. Dong, R. Socher, L.-J. Li, K. Tí, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. Tn 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248 155..55. IEEE, 2009.
L. Deng. The MNIST database of handwrilten digit images for machine learning research. JEEE Signal Processing Magazine, 2966)114 2012.
26

---

Published in Transactions on Machine Learning Research (06/2023)
T. Deselaers and V. Ferrari. Visual and semantic similarity in imagenet. In CVPR 2011, pages 1777 1784. IEEE, 2011.
A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, al. An image worth 16x16 words: Transformers for image recognition scale. arXiv preprint or Xiv:20010 10000.11199 11929, 2020.
D. Erhan, Y. Bengio, A. Courville, and P. Vincent. Visualizing higher-layer features of deep network. University of Montreal, (3):1 2009.
Fogel and D. Sagi. Gabor filters as texture discriminator. Biological cybernetics, (2):11033 113, 1989.
Gabriel. Artificial intelligence, values, and alignment. Minds and machines, 30(3):411-43) 2020.
K. Gade, C. Geyik, K. Kenthapadi, V. Mithal, and A. Taly, Explainable Al in industry. In Proceedings of the 25th ACM SIGKDD International Conference on knowledge discovery & data mining, pages 9203-3204, 2019.
S. Gautam, M. M.-C Höhne, S. Hansen, R. Jenssen, and M. Kampffimeyer. This looks like that: Enhancing self-expaanning models by prototypical relevance propagation. arXiv preprint arxiv:2108. 12204, 2021.
S. Gantam, A. Boubekki, S. Hansen, S. Salahuddin, R. Jenssen, M. Höhne, and M. Kampffmeyer. Protovae: A trustworthy self-explainable prototypical variational model. Advances in Neural Information Processing Systems, 35:17940-1777 179522, 2022a..
S. Gaulam, M. M.-. Höhne, S. Hansen, R. Jenssen, and M. Kampffmeyer. Demonstrating the risk of imbalanced datasets chest x-ray image-based diagnostics by prototypical relevance propagation. In 2022 IEEE 19th International Symposium on Biomedical Imaging (TSBI), pages 1-5, 2022b. doi:
R. Geirhos, P. Rubisch, C. Michaelis, M. Bethge, F. A. Wichmann, and W. Brendel. ImageNet-trainee CNNs are biased towards texture; increasing shape bias improves accuracy and robustness. aiv preprint arXiv:1811 19211221,,,,,, 2018.
R. Geirhos, J.-H. Jacobsen, Michaelis, R. Zemel, W. Brendel, M. Bethge, and F. A. Wichmann. Shortcut learning in deep neural networks. Nature Machine Intellgence, 2(11):665 673, 2020.
X. Glorot, A. Bordes, and Y. Bengio. Deep sparse rectifier neural networks. In Proceedings of the fourteenth international conference on Artificial Intelligence and Statistics, pages 315-323, JMLR Workshop and Conference Proceedings, 2011.
G. Goh, N. Cammarata, C. Voss, S. Carter, M. Petrov, L. Schubert, A. Radford, and C. Olah. Multimodal neurons in artificial neural networks. Distill, 6(3)xe30, 2021.
D. Grinwald, K. Bykov, S. Nakajima, and M. M.-. Höhne. Visualizing the diversity of representations learned by bayesian neural networks. arXiv preprint ar arXiv:2201.10854 2022.
T. Gu, B. Dolan-Gavitt, and S. Garg. Badnets: Identifying vulnerabilities in the machine learning model supply chain. arXiv preprind (al:1708.06733 2017.
R. Guidotti. Evaluating local explanation methods on ground truth. Artificial Intelligence, 291:103128,, 2021.
R. Guidotti, A. Monreale, S. Ruggieri, F. Turini, F. Giannotti, and D. Pedreschi. A survey of methods for explaining black box models. ACM computing surveys (CSUR), (1()):1-42, 2018.
Guyon and A. Elisseeff. An introduction to variable and feature selection. Journal of Machine Learning Research, (Mar):1157 1182, 2003.
D. Hardoon, S. Szedmak, and J. Shawe-Taylor. Canonical Correlation Analysis: An overview with application to learning methods. Neural computation, 16:2639 64, 2005. doi: 10.1162/089997666222221333
He, X. Zhang, Ren, and Sun. Deep residual learning for image recognition. In Proceedings of the IEFF Conference Computer Vision ond Pattern Recognition, pages 770-778, 2016.
A. Hedström, L. Weber, D. Bareeva, F. Motzkus, W. Samek, S. Lapuschkin, and M. M.-C. Höhne. Quantus: explainable Al toolkit for responsible evaluation of neural network explanations. arXiv preprint X20:22202.06611, 2022.
27

---

Published in Transactions on Machine Learning Research (06/2023)
E. Hernandez, S. Schwellmann, D. Bau, T. Bagashvilli, A. Torralba, and J. Andreas. Natural language descriptions of deep visual features. In International Conference on Learning Represeniations, 2021.
R. D. Hjelm, A. Fedorov, S. Lavoie- Marchildon, K. Grewal, P. Bachman, A. Trischler, and Y. Bengio. Learning deep representations by mutual information estimation and maximization. arXiv preprint arXiv:1808. 066000 2018.
G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger. Densely Connected Convolutional Networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4700-4708, 2017.
F. N. landola, S. Han, M. W. Moskewicz, K. Ashraf, W. J. Dally, and K. Keutzer. Squeezenet: Alexnet-level accuracy with 50x fewer parameters and< 0.5 mb model size. arXiv preprint 1XX::6602.... 07360, 2016.
J. Irvin, P. Rajpurkar, M. Ko, Y. Yu, S. Giurea-llcus, C. Chute, H. Markhund, B. Haghgoo, R. Ball, K. Shpanskaya, J. Seekins, D. Mong, S. Halabi, J. Sandberg, R. Jones, D. Larson, C. Langlotz, B. Patel, M. Lungren, and A. Ng. Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison. Proceedings of the AAAI Conference on Artificial Intelligence, 33:590-597, 597, 07 2019.
P. Izmailov, P. Kirichenko, N. Gruver, and A. G. Wilson. On feature learning in the presence of spurious correlations. arXiv preprint arxiv:2200. 1869, 2022.
P. Jackson. Introduction to expert systems. URL https://www.oor wwwp.wosti.ggg.. wtt.g//bbbbbloo//5111/999 7 [Accessed 16-Feb- Fe-Feb20223)
A. Jaiswal, A. R. Babu, M. Z. Zadeh, D. Banerjee, and F. Makedon. A survey contrastive self-superviee learning. Technologies, 91):2, 2020.
Jiang and O. Nachum. Identifying and correcting label bias machine learning. In S. Chiappa and R. Calandra, editors, Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, volume 108 of Proceedings of Machine Learning Research, pages 702-712. PMLR, 26 28 Aug 2020. URL. htttpa://proceedinge mlr. ppwps/v10// jiang20a. html
I. T. Jolliffe and J. Cadima. Principal component analysis: review and recent developments. Philosophical transactions of the royal sociely A: Malhematical, Plysicul and Engineering Sciences, 372(2265))20110202,, 2016.
S. Korublith, M. Norouzi, H. Lee, and G. Hinton. Similarity of neural network representations revisited. In International Conference on Machine Learning, pages 3519-3529. PMLR, 2019.
H.P. Kriegel, M. Schubert, and Zimek. Angle- outlier detection in high-diimensionl data. In Proceedings of the 14th ACM SIGKDD International Conference on Knowledge discovery and data mining, pages 444-452, 2008
A. Krizhevsky. Learning multiple layers of features from tiny images. pages 32-33, 2009. URL https://www. cs
toronto. toronto.edu//kree wttp://ariz//eaamnng---fff featare--ttuuu-u---eeee--- TR. -d.
A. Krizhevsky, Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. Communications of the ACM, 60(6):84-90, 2017.
A. Laakso. Content and cluster analysis: Assessing representational similarity in neural systems. Philosophical Psychology, 13, 05 2000. doi: 10.1080/0955 005100000055000000222666
S. Lapuschkin, A. Binder, G. Montavon, K.-R. Muller, and W. Samek. Analyzing classifiers: Fisher vectors and deep neural networks. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 2912-2920, 2016.
S. Lapuschkin, S. Wäldehen, A. Binder, G. Montavon, W. Samek, and K.-R. Müller. Unmasking clever hans predictors and assessing what machines really learn. Nature communications, 10:1096, 2019.
A. Lazarevic and V. Kumar. Feature bagging for outlier delection. In Proceedings of the eleventh ACM SIGKDD International Conference on Knowledge discovery in data mining, pages 157-166, 2005.
M. Le and S. Kayal. Revisiting edge detection in convolutional neural networks. In 2021 International Joint Conference on Neural Networks (IJCNN), pages 1-9. IEEE, 2021.
Y. Le and X. Yang. Tiny imagenet visual recognition challenge. Stanford CS 231N, 7(7):3, 2015.
C. Leacock and M. Chodorow. Combining local context and wordnet similarity for word sense identification. WordNet: An electronic lexical database, 49(2):265 283, 1998.
28

---

Published in Transactions on Machine Learning Research (06/2023)
Y. LeCun and I. Misra. Self-supervised learning: The dark matter of intelligence, 2021. URL https://ai facebook com/blog/seli supervised-learning -he-darr--atter--F wwweer--f-iinnllliieee intelligence/ [Accessed 08-Jan-2023)
Y. Li, J. Yosinski, J. Clune, H. Lipson, and J. Hoperoft. Convergent learning: Do different neural networks learn the same representations? arXio preprint 1015:2311.,7543, 2015.
Z. Li, I. Extimov, A. Gordo, C. Hazirbas, T. Hassmer, C. C. Ferrer, C. Xu, and M. Ibrahim. A whac-a-mole dilemma: Shortcuts come multiples where mitigating one amplifies others, 2022.
F. T. Liu, K. M. Ting, and Z.-H. Zhou. Isolation Forest. In 2008 8-th IEEE International Conference on Data Mining, pages 413-422. IEEE, 2008.
Ma, X. Zhang, H.-T. Zheng, and Sun. Shufflenet v2: Practical guidelines for efficient emn architecture design. Proceedings of the European Conference Computer Vision (ECCV), pages 116 131, 2018
N. Mantel. The detection of disease clustering and a generalized regression approach. Cancer Res., 27:175 17:175-178, 1967.
S. Marcel and Y. Rodrignez. Torchvision the machine-vision package of torch. In Proceedings of the 18th ACM International Conference on Multimedia, pages 1485-1488, 2010.
D. Marr and H. K. Nishihara. Representation and recognition of the spatial organization of three-dimensional shapes. Proceedings of the Royal Society of London. Series B. Biological Sciences, 200(1140):269-94. 1978.
L. McInnes, J. Healy, N. Saul, and L. Grossberger. Umap: Uniform manifold approximation and projection. Journal of Open Source Software, 3:861, 09 2018. doi: 10.21105/joss.00865
T. Mikolov, K. Chen, G. Corrado, and J. Dean. Efficient estimation of word representations in vector space. arXio preprint 1rXi::1301.378. 2013.
G. A. Miller. Wordnet: a lexical database for english. Communications of the ACM, 38(11):39 41, 1995.
G. Montavon, M. L. Braun, and K.-R. Müller. Kernel analysis of deep networks. Journal of Machine Learning Research, 12))), 2011.
G. Montavon, W. Samek, and K.-.. Müller. Methods for interpreting and understanding deep neural networks. Digital Signal Processing, 73:1-15, 2018
A. Morcos, M. Raghu, and S. Bengio. Insights on representational similarity in neural networks with canonical correlation. Advances in Nearal Information Processing Systems, 31, 2018.
A. Mordvintsev, N. Pezzolti, L. Schubert, and C. Olah. Disferentiable image paramelerizations. Distill, 3(7) 2018
J. Mu and J. Andreas. Compositional explanations of neurons. Advances in Neural Information Processing Systems, 33:17158-17163, 2020.
L. Muttenthaler, J. Dippel, L. Linhardt, R. A. Vandermeulen, and S. Kornblith. Human alignment of neural network representations, arXiv preprint ar xtv:22211.00800 2022.
A. Nguyen, Dosovitskly, Yosinski, Brox, and Clune. Synthesizing the preferred imputs for neurons in neural networks via deep generator networks, In Advences in Neural Information Processing Systems, pages 3387-3395, 2016.
A. Nguyen, J. Yosinski, and J. Clune. Understanding neural networks via feature visualization: survey. In Explainable AT: interpreting, explaining and visualizing deep learning, pages 55 76. Springer, 2019.
M. Nguyen, J. Yosinski, and J. Clune. Innovation engines: Automated creativity and improved stochastic optimization via deep learning. In Proceedings of the 2015 annual conference genetic and evolutionary computation, pages 959-966, 2015.
T. Nguyen, M. Raghu, and S. Kornblith. Do wide and deep networks learn the same things? Uncovering how neural network representations vary with width and depth. arXiv preprint arXio:2010. 15327, 2020.
T. Nguyen, M. Raghu, and S. Korublith. On the origins of the block structure phenomenon in neural network representations. arxiv preprint ar Xrx:2220000188,, 2022.
29

---

Published in Transactions on Machine Learning Research (06/2023)
C. Olah, A. Mordvintsev, and L. Schubert. Feature visualization. Distilt, 2(11) 2017.
C. Olah, N. Cammarata, C. Voss, L. Schubert, and G. Goh. Naturally occurring equivariance neural networks. Distill, 5(12):e00024 004, 2020.
D. Omeiza, S. Speakman, C. Cintas, and K. Weldermariam. Smooth Grad Grad-Cam++:: An enhanced inference level visualization technique for deep convolutional neural network models. arXiv preprint arXim:1908.0122, 2019.
T. Pedersen, S. Patwardhan, J. Michelizzi, et al. Wordnet: Similarity- the relatedness of concepts. In AAAI, volume 4, pages 25 29, 2004.
X. Qin and Z. Wang. Nasnet: A neuron attention stage-by-stage net for single image deraining. ar preprind arxi: 1912. 2.1111, 2019.
A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervision. In International Conference Machine Learning, pages 8748- 8763. PMLRR 2021.
M. Raghu, J. Gilmer, J. Yosinski, and J. Sohl-Dickstein. SVCCA: Singular Vector Canonical Correlation Analysis for deep understanding and improvement. Xiviv preprint arXiv: (aoo:/006000006 2017.
M. Raghu, T. Unterthiner, Kornblith, C. Zhang, and A. Dosovitskiy. Do vision transformers see like convolutional neural networks? Advances in Neural Information Processing Systems, 34:12116-12125
2021.
J. Ramsay, Berge, and G. Styan. Matrix correlation. Psychometrika, 49:403-424 423, 09 1984 doi: D..100/BFF2300029.
R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. High-resolution image synthesis with latent diffusion models. In Proczedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition (CVPR), pages 10684-10695, June 2022.
C. Rudin. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature machine intelligence, 1(5):206 215,, 2019.
L. Ruff, J. R. Kauffmann, R. A. Vandermeulen, G. Montavon, W. Samek, M. Kloft, T, G. Dietterich, and K.-R. Müller, A unifying review of deep and shallow anomaly detection. Proceedings of the IEEE, 109(5):756 795177666799997 2021.
O. Russakovsky, Deng, Su, J. Krause, S. Satheesh, S. Ma, Huang, A. Karpathy, A. Khosla, Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115:211-28 12222, 2015.
W. Samek, A. Binder, G. on, S. Lapuschkin, and K.-R. Müller. Evaluating the visualization of what a deep neural network has learned. IEEE transactions on Neural Networks and Learning Systems, 28(11):26602286) 2676, 2016.
W. Samek, G. Montavon, A. Vedaldi, L. K. Hansen, and K.RR. Müller. Eaplainable AI: interpreting, eaplaining and visualising deep learning, volume 11700. Springer Nature, 2019.
W. Samek, G. Montavon, S. Lapuschkin, C. J. Anders, and K.RR. Müller. Explaining deep neural networks and beyond: A review of methods and applications. Proceedings of the IEEE, (0943):247-278, 2021.
M. Sandler, A. Howard, M. 7hu, A. Zhmoginov, and L.-C. Chen. Mobilenetv2: Inverted residuals and linear bottlenecks.
In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 4510-4520, 2018.
B. Schölkopf, J. C. Platt, J. Shawe-Taylor, A. J. Smola, and R. C. Williamson. Estimating the support of high-dimensional distribution. Neural computation, 13(7):1443-1471, 2001.
P. Schramowski, W. Stammer, S. Teso, A. Brugger, F. Herbert, X. Shao, H.-G. Luigs, A.-K. Mahlein, and K. Kersting. Making deep neural networks right for the right scientific reasons by interacting with their explanations. Nature Machine Intelligence, 2(8):476-486, 2020.
R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. International Journal of Computer Vision, 128(2):336359, 10 2019. ISSN 1573-1405. doi: 10.1007/s1 121111163--901222000
K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recoguition. arXiv preprinl Xix: 1409. 1556, 2014.
30

---

Published in Transactions on Machine Learning Research (06/2023)
D. Smilkov, N. Thorat, B. Kim, F. Viégas, and M. Watlenberg. Smoothgrad: removing noise by adding noise. arXio preprint X w0w. 03825, 2017.
K. O. Stanley, Compositional pattern producing networks: A novel abstraction of development. Genelic programaming and evoleable machines, 8:131 162, 2007.
M. Sundararajan, A. Taly, and Q. Yan. Axiomatic attribution for deep networks. In International Conference on Machine Learning, pages 3319-3328. PMLR, 2017.
C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, Goodfellow, and R. Fergus. Intriguing properties of neural networks. arXiv preprint arXiv:1812.619 6199, 2013.
C. Szegedy, V. Vanhoucke, S. loffe, J. Shlens, and Z. Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE Conference Computer Vision and Pattern Recogrittin,, pages 2818 2826, 2016.
B. Thomee, D. A. Shamma, G. Friedland, B. Flizalde, K. Ni, D. Poland, D. Borth, and L..J. Li. Yfcc100m: The new data in multimedia research. Communiicatios of the ACM, 59(2):64 73, 2016.
I. Tiddi et al. Directions for explainable knowledge-enabled systems. Knowledge Grophs for eXploinable Artificial intelligence: Foundations Applications and Challenges, 47:245, 2020.
B. Tran, J. Li, and A. Madry. Spectral signatures in backdoor attacks. Advances in Neural Information Processing Systems, 31, 2018
F. Trozzi, X. Wang, and P. Tho. Umap as a dimnensionality reduction tool for molecular dynamics simulations of biomacromolecules. comparison study, The Journal of Physical Chemistry B, 125(19):5022 5034, 2021.
L. Van der Maaten and G. Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008
M. M.-. Vidovic, N. Görnitz, K.-R. Mūller, G. Rätsch, and M. Kloft. Opening the black box: Revealing interpretable sequence motifs kernel-based learning algorithms. Joint European Conference on Machine Learning and Knowledge Discovery in Dotaboses, pages 137-153. Springer, 2015.
M. M.C. Vidovic, N. Görnitz, K.-R. Müller, and M. Kloft. Feature importance measure for non-linear learning algorithms. arXiu preprint orXiv: 07567, 2016.
D. Wallis and 1. Buvat. Clever hans effect found in widely used brain tumour mri dataset. Medical Image Analysis, 77:102368, 2022.
X. Wang, Y. Wn, A. Zhang, X. He, and T.SS. Chua. Towards multi-grained explainability for graph neural networks. Advances in Neural Information Processing Systems, 34:18446-11 18458, 2021.
Y. Wang. CIFAR-100 Resnet PyTorch 75.17% Accuracy taggle.com. https: :/www www.w.aagg...om how.e//ccoc/co//e/eeeeeenaga 2021. [Accessed 08-Jan-2023)
Weiss, M. Khoshgoftaar, and D. Wang. survey of transfer learning. Journal of Big data, 3(1):1-40, 2016.
R. Wightman. Pytorch image models.
hag.:aaaaaaappoocr-a---aaaeooee models 2019.
D. Wu, J. Y. Poh Sheng, G. T. Su-En, M. Chevrier, J. L. Jie Hua, T. L. Kiat Hon, and J. Chen. Comparison between umap and t-sne for multiplex-immunnfuorreeees derived single-cell data from tissue sections. BioRaiv, page 5-19659, 2019.
7. Wu and M. Palmer. Verb semantics and lexical selection. arXio preprint cmp-lg/9406033 1994.
Xiao, L. Engstrom, A. Tlyas, and A. Madry. Noise or signal: The role of image backgrounds in object recognition. arXiv preprint ar Xiv:20066 09994, 2020.
F. Xu, H. Uszkoreit, Y. Du, W. Fan, D. Zhao, and J. Zhu. Explainable Al: brief survey on history, research areas, approaches and challenges. In CCF International Conference on natural language processing and Chinese computing, pages 574. Springer, 2019.
Z. Yuan, Y. Yan, M. Sonka, and T. Yang. Large-scale Robust Deep AUC Muximization: A new surrogate loss and empirical studies medical image classification. pages 3020 2020-3029, 10 2021. doi: (0.1009/1CC448922.022 .03003.
31

---

Published in Transactions on Machine Learning Research (06/2023)
J. R. Zech, M. A. Badgeley, M. Liu, A. B. Costa, J. J. Titano, and E. K. Oermann. Variable generalization performance of a deep learning model to detect pueumonia in chest radiographs: a cross-soctional study. PLoS medicine, 15(11): e1002683, 2018
M. D. Zeiler and R. Fergus. Visualizing and understanding convolutional networks. In European Conference on Computer Vision, pages 818-833. Springer, 2014.
F. Zhuang, Z. Qi. K. Duan, D. Xi, Y. Zhu, H. Zhu, H. Xiong, and Q. He. A comprehensive survey transfer learning. Proceedings of the IEEE, 109(1):43 76, 2020.
32

---

Published in Transactions on Machine Learning Research (06/2023)
A
Appendix
A.1 Evaluation
In the evaluation, two datasets were used: ILSVRC2012 (ImageNet 2012) Deng et al. (0099) and CIFAR- 100
Krizhevsky (2009). For ImageNet, we employed eight different braitrained models: ResNet18 He et al.
2016
AlexNet Krizhevsky et al. (2017), Inception V3 Szegedy et al. (2016), DenseNet 161 Huang et al. (2017) J
[andler al. (2018). ShuffleNet. V2 Ma et. al. (2018), obtained from the torchvision-modelly package Marcel and Rodriguez (2010), as well ViT Dosovisskk et al. (2020) and BETT Krizhevsky et al.
MobileNet V2
2017 obtained from the viforrhhvisionmmodee
library Wightman 2019). For the CIFAR-100 datasct,
we trained seven networks: ResNet 18, MobileNet V2, ShuffleNet VI, and V2, NASNet Qin and Wang (0019),
SqueeeNet Tandola et al. (2016), and VGG 11 Simonyan and Zisserman (2014), using the Pytorch-cifards
GitHub repository git (2020), while the ResNet9 network was trained using a publicly available Kaggle
notebook Wang (2021).
The semantic baseline distances between concepts for both datasets obtained using the NLTK package Bird et al. (2009). There is a cross-connend cons-connection between class labels and WordNet entities for ILSVRC2012, as the classes are inherently connected with WordNet synsets. For CIFAR-100, labels to synsets by matching class label names with WordNet synset names, For 98 classes, WordNet synsets were found. For the remaining two classes, "aquarium fish" and "maple tree", WordNet synsets for "fish" and "maple" were used, respectively, due to the absence direct name match.
manually connected the
For the Werd2Vec distance, we used the WordNet synset name as the textual label. If textual label contained multiple words, the distance between two classes was determined as the maximum distance among all possible word pairs between the textual labels of the two classes.
A.2 Experiments
A.2.1 Probing dataset
CH-class
AULLE
Normal-class
Figure 19: Illustration of the Probing Dataset. The figure depicts images from the probing dataset utilized evaluate the representation's capacity to distinguish between watermarked (CH) and non-watermarken (normal) images. The watermarked class images are identical the normal class images, except for the addition of a random test string at random location the image.
To assess the ability of the identified representations detect undesirable concepts, we created two probing datasets for the binary classification of Chinese and Latin text detection. We modified one class of images by adding specific watermarks while leaving the other class unchanged. We used baseline dataset of 998
33

---

Published in Transactions on Machine Learning Research (06/2023)
ImageNet images 1 to create 2 probing datasets (Chinese and Latin) by inserting random textual watermarks, as shown in Figure 19 For the Chinese characters detection problem, the watermarks were generated by randomly selecting 7 out of the 20 most commonly used Chinese characters Da (2004), and a similar process was followed using the English alphabet for the Latin text detection problem. The font size for all watermarks has been set to 30, while the image dimensions remain standard al 224 x 224 pixels. AUC ROC was used as the performance metric to evaluate the representations' ability to differentiate between watermarked and normal classes. The true labels provided by the two datasets were used, where class 1 represents images with watermark and class 0 represents images without. We computed the scalar activations for all images from both classes for a specific neural representation and then calculated the AUC ROC classification score based on the differences in activations using the binary labels. A score of indicates a perfect classifier, consistently ranking watermarked images higher than normal ones, while a score of 0.5 indicates a random classifier.
A.2.2 ImageNet ResNet18
In the following, we provide additional details on the ResNet18 He et al. (0016) experiment, discussed in the main paper. The model was downloaded from the Torchvision library Marcel and Rodriguez (2010) and s-AMS were generated with parameters 5 and m 500 using the DORA package.
Figure 20 illustrates the cluster of reported representations in the average pooling layer of the model, specifically neurons 154, 129, 347, 489, 81, 439, and 282, along with the sensitivity of other neurons to Chinese watermarks. It can be seen that representations close to the reported cluster also exhibit sensitivity towards malicious concepts. For additional context, Figure shoss the natural Octivation-Maxiimsst signals (n-AMS) for the reported representations, obtained using 1 million subsamples of the ImageNet 2012 train dataset. The presence of Chinese watermarks in the n-AMS further supports our hypothesis of the Clever-Hans nature of these reprosentations.
To examine which output class logits may be compromised by Clever-Hans (CH) behavior, we used the AMA of the reported neurons to obtain class predictions on these signals. Figure 26 shows several A-MMS for the reported reprosentations along with the network's predictions for the corresponding data points. We observod that cerlain classes, such as "carton" (478), "apron" (411), "swab, swob, mop" (840), "monitor" (664), and "broom" (462) were frequently predicted with high scores. When we computed the s-AMS for selected output logits, we found similar Chinese patterns, similar to those observed in the reported neurons of the average pooling layer (see Figure 26). These results suggest that such artifacts learned by the network pose a potential threat to applications due to the network's 's tendency to classify images with added watermarks as belonging to one of these classes.
A.2.3 ImageNet DenseNet121
The DORA framework was employed to investigate the pre-trained DenseNet121 on the ImageNet dataset Huang et al. (2017). Specifically, attention was focused on the last layer of the feature extractor, which comprised 1024 channel representations. The study primarily examined two outliers detected by DORA: neuron 768 and neuron 427, along with some of their nearest neighbors in the EA distance. Following an analysis of the s-AMS for both neurons, specific symbolic patterns were observed, which were characteristic of character detectors. Neuron 768 was identified as a Chinese character detector, while neuron 427 was identified as a Latin text. detector. Figure [6 in the main paper and Figure 22 depict these neurons, along with their closest neighbors in EA distance, which exhibited similar properties. The hypothesis was further supported by visualizing the n-AMS across the ImageNet dataset, as demonstrated in Figure 23
As mentioned in Section 6.1 we find that the outliers found by DORA are maintained during fine-tuning on another dataset, e.g. the CheXpert challenge. The CheXpert challenge benchmarks various deep learning models on the task of classifying multilabel chest radiographs and additionally provides human experts, e.g. radiologists, with performance metrics for comparison. The data set itself consists of 224,316 training, 200 validation, and 500 test data points. The current best approach in terms of AUC-ROC score uses an ensemble of five DenseNet121's Huang et al. (017)) that were pre-trained on the ImageNet dataset and fine-tuned
Images were obtained from bttps://githhbl cot/ wtt.iwaattt/iaaaneeeeeetttteeeee sample---geeeeeeeeeeeeee with the exception of two images (of the class "carton" and 'terrapin)) that already exhibit watermarks.
34

---

Published in Transactions on Machine Learning Research (06/2023)
Reported CH-cluster
AUC ROC
Reported LOF outliers (p 0.01,
Figure 20: Detailed illustration of the cluster of malicious representations found. All of the figures illustrate the representation atlas of the average pooling layer of ResNet18, calculated using the DORA distance metric. From left to right: illustration of the reported Chinese detector cluster, the sensitivity of different representations for detecting Chinese watermarks, and set of reported outliers among the representations using the LOF method. From the middle figure, it can be observed that the cluster of reported representations exhibits high sensitivity towards the artifactual concept of the desired task, and the closer the representations are to the cluster in the representation atlas, the more they are able to detect malicious concepts in the data.
Original #768
Finetuned # 768
The Posiivv Rate
Pre-trained, #768, 0.98 Finetuned, #768, AUC 0.66 Average
False Positive Rate
Figure 21: Survived Chinese-characters detector. Neuron 768 learns to detect Chinese logographic symbols during pre-training (top left) and does not unlearn this behavior during fine-tuning on the CheXpert dataset (top right). The AUC values of the neurons' activation on images corrupted with Chinese watermarks are still high after pre- trainiin
35

---

Published in Transactions on Machine Learning Research (06/2023)
768
776
Receiver Ope wting Characterite abc
773
614
748, ALC 2098
38, ABLUU--
278, AUC 233
AUC
Autrage
CNAP:
Fobe Pesttr Fate
Figure 22: DenseNet121 - Chinese-characters detector. Applying DORA to the last layer of the feature extractor of DenseNet121 yields, among others, Neuron 768, which corresponds to the upper left of the 6 feature visualizations. From Neuron 768 as well as from its five closest neighbors (shown left), we can observe semantic concepts resembling Chinese logograms. The AUC values were computed using the channel activations on a data set that was corrupted with watermarks written in Chinese. As shown, the AUCs are high for the representation outliers found by DORA, compared most of the other representations, which indicates that they indeed learned to detect Chinese logograms.
Maximally Activating Images for Neuron 768
Maximally Activating Images for Neuron 427
pretranss DerSetettz.. fesures 1p9a
prttinned DerSeWet22. fesures
Faq
SUE FUNDIONA
SUBSCRBER
8.5 CM
(a) Neuron
(b) Neuron 427
Figure 23: Maximally Activating images for different neurons in DenseNet121. Illustration of the top 15 images that trigger the highest activations for the Chinese watermark detector (neuron 768) and the Latin text detector (neuron 427) in the 'Teatures' layer of DenseNet121.
by optimizing special surrogate loss for the AUC-ROC score Yuan al. 2002 The training code can be found in this public repository https: //t::////ttttbbuu wwwp/pppiiiiaaiiiooAAA//Aaaa one 1en1eNet121 using this approach a downsampled version of the CheXpert data with resolution of 256x256x3. The converged model yields an AUC-ROC score of 87.93% on the validation dataset. Having the finetuned DenseNet 121 and the outlier neuron 768 at hand we show the Feature Visualizations and the AUC-ROC curves for both the pre-trained and fine-tuned channel on an ImageNet subset with both uncoruppted and corrupted images with Chinese watermarks in Figure 21
We choose to finetune
A.2.4 CLIP ResNet 50
The s-MMS for the CLIP ResNet was computed using the same parameters as Goh et al. (2021) with the Lucent library. The number of optimization steps was set to 512. The analysis was conducted representations (channels) from the *layer 4" layer of the model. (Details on the SAMSS generation
36

---

Bisong and Published in Transactions on Machine Learning Research (06/2023)
Table 4: Clusters of CLIP "layer4" representations. This table presents several interesting clusters and the indexes of the corresponding representations that examined through manual inspection of the s-AMS and most activating images.
Cluster
Representations
Explicit Pocleiit/oonnn Pornographic Moncy/Finance
95, 255, 996, 1502, 2011
785, 1376, 1731
230, 250, 417, 521, 652, 654, 694, 1008, 1234, 1301, 1340,
Reptiles
1364, 1445, 1598
Fish/ Aquarium
1193, 1384
13. 165, 235, 536, 780, 931, 1037, 1261, 1247, 1423,
Asia ssog-geogaaphic
1669, 1761, 1874, 1898
parameters can be found at http::://itttb..comm/////
http://t..ccooppaaaaaaaPaaarrrrr and Lucent library at https /ggithubccom/reee hwtp./////eeeetfrap////eeee
Star Wars representation
Figure 3 shows the limitations of the n-AMS approach when the data corpus for analysis differs from the training datasct. Figure 27 further illustrates n-AMS collected from ImageNet and Yahoo Creative Commons
Thomee et al. (2016) datasets via OpenAI Microscope. Text Feature Visualization Goh et al. 2021 supports our hypothesis that the model detector of Star Wars-related concepts.
Outlier representations
Analysis of the representations space of the CLIP model yielded number of potential candidates to be considered outlier representations, namely neurons 631, 658, 838, 1666, 1865, and 1896. In Figure 28 we illustrate 3 s-AMS signals, alongside n-AMS images, collected from the ImageNet dataset per each reported representation, collected using OpenAI Microscope. While it is hard to explain the anomalous nature of neurons 631, 658, 838, 1666, and 1896, we can clearly observe how different the concept of neuron 1865
Clusters of representations
We manually examined several distinctive classes of representations in "layer4" of the CLIP model after computing the representation atlas for the channel representations. Table summarizes the results of our analysis and shows interesting clusters found along with the associated neurons. Figure 29 shows synthetic and natural AMS, providing evidence for the assignment of neurons to their respective clusters.
A.3 Experimental setup
All described experiments, if not stated otherwise, were performed on the Google Colab Pro Bisong 2019) cnvironment with the GPU accelerator.
37

---

Published in Transactions on Machine Learning Research (06/2023)
Maximally y Activating Images Neuron 154
Maximally Activating Images for Neuron 347
(a) Neuron 154
(b) Neuron 347
Maximally Images for Neuron 129
Maximally Images for Neuron 489 Sedlvilh soange
Resieria aivrage
(c) Neuron 129
(d) Neuron 489
Maximally Activating Images for Neuran 8.
Maximally Activating Images Neuron 282
Bealletin evitagg
Neuron 81
(f) Neuron 282
Figure 24: Maximally Activating Images for different ResNet18 neurons, reported in the cluster of malicious representations. The figure shows the 15 highest Activating Images for various neurons in the "avgpool" layer of the ResNet18 network, which were identified as being in the cluster of malicious representations. The signals were calculated using subset of million images from the ImageNet 2012 training dataset. It can be observed that among the top activating images, there are images of Chinese watermarks, supporting the hypothesis that these neurons have learned undesirable concepts.
38

---

Published in Transactions on Machine Learning Research (06/2023)
Neuron 154
carton apron safe
0.991 carton
0.942 carton
0.675 carton
0.993 carton
0.923
0.005 safe
0.032 apron
0.323 safe
0.005 apron
0.076
0.004 apron
0.02 safe
0.001 apron
0.001 safe
0.001
Neuron 347
candle, taper whistle monitor
0.76 candle,
0.936 monitor
0.515 whistle leaf
0.384 swab,
0.912
0.206 whistle
0.063 swab, swob
0.287 beetle
0.352 whistle
0.083
0.028 safe
0.001 candle, taper
0.153 emmet
0.158 monitor
0.005
Neuron 129
broom
1.0 broom
0.918 broom
1.0 broom
0.962 broom
0.999
swab, swob safe
0.0 swab, swob
0.082 swab, swob
0.0
0.025 swab, swob
0.001
0.0 safe
0.0 safe
0.0 apron
0.006 safe
0.0
Neuron 489
toilet tissue
0.992 toilet tissue
0.998 toilet tissue
toilet tissue
0.662 toilet tissue,
0.773
paper towel
0.004 paper towel
0.001 paper towel
0.0 bath towel
0.28 bath towel
0.119
gazelle
0.002 bath towel
0.001 bath towel
bucket, pail
0.023 hartebeest
0.083
Figure 25: S-AMS and model predictions for reported neurons in ResNet18. Figure illustrates the AAMS signals for four different reported neurons in the average pooling layer of ImageNNe-trainedd ResNet18, along with the model's predictions for the top three classes with their respective softmax scores.
39

---

Published in Transactions on Machine Learning Research (06/2023)
Logit 478
"carton"
Logit 411 "apron"
Logit 840 "swab, swob" "
Logit 664 "monitor"
Logit 462 "broom" "
Figure 26: S-AMS for several ResNet18 logits. Figure shows s-MS for the output logit representations of ResNet18. Similar to the reported neurons from the average pooling layer, the logits display logographic patterns, logographic patterns specific to Chinese character detectors, suggesting that these classes may be particularly affected by CH behavior.
40

---

Published in Transactions on Machine Learning Research (06/2023)
UNI766
UnIt 264
FENTURG VESONLIDATIO
DNTAART SAMPLES
maggu
Uiees coverit sheee
VISUNUSATION
showlind
Placac of irages from the traniing theanet that
product with reutin
Text Pat mac omeesE
0:00 highighting
030 hor highiigting
0.71 lghosttr paths dhoskind
lines but wherr somvon daa lavo sypland.
but when
0.00 suamerr fyrtheber
resull in the begood
0.09 rachiren ion nosations in ladocca laser ueopon one lirnsaber S
actisttioss torm the glown
0.20 of right ghiing lot point has 36 Kjccicacome
0.29 notast ona al jucoka love weupor wightawerr
0.27 High highting bou pailict
when som cone oolo
0.27 HeMighhtne the para conalll hes comoone aoko you drew
dau lugs for
ligimaaeer
CC OPrie dnsouaee
me high ghiing
someon
ky Iyrbaeerr wring
lasoon poinecr
0.22 stoe
0271 ghhtoom
charly mashomegentung
0.10 ditarce
efter juputer
0.17 symsoom lawer refusing chanl mad hur
0.16 buter espn membie
0.15 rempoint frie becomee
Figure 27: CLIP ResNet Neuron 744. The figure shows A-MSS and Maximally Activating Images for neuron 744 in the "layer 4" layer of the model, computed for 2 different data corpora. The observed signals and explanations from Text Feature Visualization confirm that the neuron can detect Star Wars-related concepts. Results obtained from OpenAI Microscope.
41

---

Published in Transactions on Machine Learning Research (06/2023)
CLIP ResNet 18, Layer 4, OpenÅl Microscope
1865
CLIP ResNet 18, Layer 4, OpenAl Microscape
1896
S-AMS
S-AMS
ImageNet Images
ImageNet In--maaes
CLIP ResNet 18, Layer 4, OpenAl Microscope
1666
CLIP ResNet Layer neuron 838 OpenAl Microscope
S-AMS
S-AMS
ImageNet images
ImageNet AM Images
UIV
CLIP ResNet Layer neuron 658 OpenAl Microscope
CLIP ResNet 18, Layer neuron 631 OpenAl Microscope
S-AMS
S-AMS
ImageNet Images
ImageNet AM Im--maass
Figure 28: S-AMS and Maximally Activating Images for reported outlier neurons. Figure illustrates AAMS and Maximally Activating Images for the reported outlier neurons in the "layer 4" layer of the CLIP ResNet 50 model, collected from OpenAI Microscope.
42

---

Published in Transactions on Machine Learning Research (06/2023)
Reptiles
Asia-geogragraphic
652
13
1340
235
1364
536
1598
931
OSHITL
1301
1037
Money/Finance
1247
785
1261
1376
1669
1731
1898
Figure 29: s-AMS and maximally activating images from ImageNet for the neurons the reported clusters. This figure shows the s-AMS and maximally activating images for representations assigned to the various reported clusters. The S-M were generated, while the maximally activating images from ImtgeNet were collected via the OpenAI Microscope. Representations of explicit pornographic content were excluded due to the presence of obscene images.
43


Reviews:
[{"review_id": "wBbLQGPwdY", "reviewer": "Reviewer_3SJw", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strength**\n- The proposed method is simple and data-independent; this means using the method does not requires access to the training data. \n\n- The idea of comparing distances between s-AMSs and their embeddings is quite interesting.\n \n- Extensive experiments on both controlled and real-world settings validate the method's effectiveness. \n\n**Weakness**\n- Not sure why s-AMS similarity is related to outlier detection, the authors provide some vague connections but more concrete connections need to be illustrated \n- The method relies on s-AMS, which is not very stable, and the problem of solving s-AMS is often tricky; this would result in bad s-AMS results, which will influence the performance of the method. (Though the reviewer does see the authors trying to generate several s-AMS and use them all) \n- Would the simple average across multiple different s-AMSs embeddings result in a non-representative embedding of s-AMSs, and make the cosine similarity meaningless? \n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper shows the self-explaining capabilities of DNNs by extracting semantic information contained in the synthetic Activation Maximisation Signals (s-AMS). Specifically, they propose to analyze the similarities of the s-AMS and further employ this information to identify outlier representations. ", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "In definition 3, the distance EA should be between $f_i$ and $f_j$, not $f_i$ and $f_q$. Not sure where is the $q$ come from?\n\nIn figure 3, the EA distance subfigure, since the EA distance matrix is $k\\ times k$, how can you get the labels on which block represents Tiny ImageNet and which block represents EMNIST? A clarification needed to be added. Is it because of calculating the logits, so they correspond to each category? \n\nAnother question related to figure 3, it is also hard to directly see from the EA distance plot that the Mini ImageNet and EMNIST have different patterns. And the distance between ImageNet and EMNIST is not very small also (that's why we see green and dark blue all over the place). Can you give an explanation for this? \n\nWhat are the dots in the representation atlas means? Do they mean different examples or still different categories, like in the EA distance figure? \n\nF is a function class, and the outputs are concatenated in the end; this is not clear unless you thoroughly read the paper.\n\nWhy is UMap used? Is there any particular reason for using it rather than other methods? since it is an important step for outlier detection, the author should justify it in details. ", "broader_impact_concerns": "This work does not bring any particular ethical concerns. ", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ecxGErOnzr", "reviewer": "Reviewer_V27b", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. The figures are quite helpful in understanding the context. For instance, Figure 2 provides a nice overview about the proposed approach and Figure 3 summarizes the overall contribution.\n\n2. The mathematical definition and the method sections are generally clear, I can understand the goal of the proposed algorithms. \n\n3. The experiments are done extensively, where the authors study several architectures including ResNet, MobileNet, and the recent large-scale model CLIP. This makes the study much convincing and successfully demonstrate the robustness against model selection.\n\n4. I like the related work sections, where the authors give a nice summarization about the overall works in neural network interpretability.\n\nWeaknesses:\n\n1. The motivation is not clear. After reading the complete draft, it seems that the main application is outlier detection. Nevertheless, the method sections start by an approach to compare the distance between classes, which make the readers a bit confused. In particular, measuring the semantic distances between dimensions and outlier detection seems a bit disconnected for me.\n\n2. Evaluating anomaly-identification capabilities: In the experiments, the authors propose to insert random representations in the network layer to evaluate the ability of outlier detection methods. However, typically this should done by evaluating out-of-distribution (OOD) samples instead of directly injecting noise to the latent space. There is no guarantee that real world OOD samples will lead to the same noise pattern, which makes this evaluation setting odd.\n\n3. Data-free claim: In the experiment of section 5.1, it seems that identifying those outlier representations still require the help of data. In this case, what makes the propose approach different from previous works such as network dissection?\n\n4. The baselines of outlier detection (OD). I believe there are numerous baselines for OD, including density estimation, noise contrastive estimation, data-based approach with binary classification, uncertainty estimation… It is not clear how and why the proposed approach is preferable compared to the previous approaches.\n\n5. The visualization: After checking Figure 6, 7, 9, 16, 20, I found the visualization of neurons not as interpretable as the claims. It is hard to parse what is the semantic meaning of the visualization.\n\n\nMinor Questions:\n\n1. Organization: It seems that section 4 and 5 are both experimental section. What is the purpose to separate them?\n\n2. In figure 12, it seems that the Chinese characters used to perturb the data is highly repeated. Same characters are being used in the same image, even across the image. How large is the dictionary for the Chinese characters?\n\n3. In page 5, F is treated as a function class, but in the last equation it takes input just like a function. After reading the whole section I understand F is treated as a concatenation of each f_i, but it does lead to some confusions here.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose DORA, a new data-agnostic framework for the analysis of the representation space of DNNs. By maximizing the response with respect to a hidden dimension, the algorithm is able to generate input images that are interpretable and account for the similarity between dimensions. In addition to similarity measure, the authors apply the proposed technique to outlier detection and show that DORA is able to detect contaminated images with watermark.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Despite the weaknesses, the overall contribution is inevitable. Some minor changes can make the paper much more clear, which include (1) making the motivation clear, (2) addressing the concerns in outlier detection, (3) revisiting the data-free claim for outlier detection.", "broader_impact_concerns": "I did not see any significant ethical concern raised in this paper, which it addresses a fundamental problem in neural network interpretability.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "CmzvMwAb7p", "reviewer": "Reviewer_fRDS", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The study of interpretability with visualization methods in deep learning is interesting.\n\nWeaknesses:\n1. Lack of a clear definition of \"outliers.\" What is the exact definition of \"outliers?\" The authors need to clarify and provide the mathematical definition of \"outliers.\"\n\n2. The novelty is limited. DORA framework contains two parts. The first part is mainly based on Feature Visualization [Olah et al. (2017)]. The second part uses UMAP dimensionality reduction algorithm [McInnes et al. (2018)] to visualize the representation space. Therefore, the novelty of this paper is limited.\n\n3. What’s the exact definition of neural representation $f$? Does $f_i$ denote the intermediate representation of the DNN or the output of the DNN? Please clarify.\n\n4. Definition 2 seems problematic. \n(1) The authors need to introduce the detailed method for generating s-AMS in Definition 2. Theoretically, if the authors do not constrain the scope of $x$, $s=argmax_x f(x)$ can not be solved.\n(2) I suggest the authors clarify why we need to maximize $f_i$. Why not select a significantly negative number for $f_i$? Please further clarify.\n\n5. In step 1 (Generation of s_AMS), $f_i \\in F$ seems problematic. $f_i \\in F$ indicates that $F$ is a set. However, $F$ contains k neural representations and should not be a set. \n\n6. The proposed EA distance metric is ad-hoc and lacks theoretical guidance. How can the authors guarantee the correctness of EA distance calculated with cosine similarity? In general, a rigorous metric should have many axioms. For example, the Shapley value [c1] is known to be the unique method that satisfies four axioms, including efficiency, symmetry, linearity, and dummy. Therefore, we consider that four axioms guarantee the correctness of the Shapley value. I suggest the authors provide axioms or properties to prove or define the correctness of EA distance metric.\n\n[c1] Grabisch, Michel; Roubens, Marc (1999). \"An axiomatic approach to the concept of interaction among players in cooperative games.\" International Journal of Game Theory. 28 (4): 547-565.\n\n7. The mathematical definition needs to be revised and clarified. It is difficult to be followed and understand. Specifically,\n(1) The mathematical definition of $x$ in the equation in Section 3 on Page 4 and Definition 2 should be defined and clarified.\n(2) The authors need to introduce $l(c_i, c_j)$ in Shortest-Path distance and Leacock-Chodorow distance in Section 4 on Page 8. \n\n8. Definition 2 seems problematic. The authors need to introduce the detailed method for generating s-AMS in Definition 2. Theoretically, if the authors do not constrain the scope of $x$, $s=argmax_x f(x)$ can not be solved.\n\n9. UMAP dimensionality reduction algorithm needs to be further introduced and discussed. Can we use traditional dimensionality reduction algorithms, such as PCA, t-sne, and spectrum cluster? I suggest the authors theoretically analyze the advantages of using UMAP. \n\n10. The average embedding operation in the first equation on Page 6 seems problematic. Visual embeddings may have different diverse directions. Simply averaging these visual embeddings may offset these semantic representations in some directions and cause the loss of semantic information. Is there any theoretical foundation?\n\n11. Why do the authors obtain different samples $s_1, s_2, s_3$? Why do you consider [Olah et al. 2017] as a convincing solution for s_AMS generation? How to theoretically prove the number of optimization steps is suitable for s_AMS generation?\n\n12. Generating s-AMS may cause problems in interpretability. Visualizations generated by DORA are not easy to be interpreted or understand. The proposed method does not necessarily explain why or how the representations are formed.\n\n13. Figure 3(middle) is confusing. According to Definition 3, each dot represents a neural representation. However, the legend in Figure 3(middle) indicates that each dot represents a sample in the dataset.\n\n14. Experiments in Section 4.1 and Section 4.2 are confusing. What is the meaning of \"ground truth\" in Section 4.1 and Section 4.2? Does the \"ground truth\" refer to the distance calculated by Shortest-Path distance or Leacock-Chodorow distance? If so, the authors should clarify why these two distances can be regarded as ground truth distances.\n\n15. The compared baseline methods (Shortest-Path distance or Leacock-Chodorow distance) are not enough. I suggest the authors compare with more methods.\n\n16. There are no labels in all formulas. I suggest the authors label all formulas.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes DORA (Data-agnOstic Representation Analysis) to analyze the representation space of DNNs. DORA can detect semantically anomalous representations without human supervision. The authors demonstrate that DORA can find artifactual representations and visually illustrate semantic similarities between representations of different samples.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please see the \"strengths and weaknesses\" section.", "broader_impact_concerns": "none", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Reviewers expressed concerns mostly regarding the presentation, the major claim and the level of details of the proposed pipeline. Authors engaged the raised concerns well and updated their work up to a satisfactory point as to reach a "leaning to acceptance" recommendation from two reviewers out of three. One reviewer, instead, has been very skeptical concerning the claims about explainability and definitions around what is an outlier and what can be truly aligned with humans need to be better formalized, and did not update their scores. 

I agree with this reviewer's concerns when it comes to tuning down claims and about the level of rigor needed to publish a journal paper. At the same time, I also believe that the updated manuscript has already reached a level of formality that i) is found in other papers dealing with representation learning, feature visualization for neural networks and  outlier detection and ii) allows to correctly understand what is proposed and reproduce it. Furthermore, I believe that the work goes in an interesting direction, and the yet preliminary pipeline can be of enough interest for the community to build on it. 

The paper is therefore accepted with minor revision.

The authors are asked to incorporate the following modifications in the new revision:

- Amend/simplify abstract and introduction w.r.t. "self-explaining capabilities within the network", as there are no inherent self-explainations
- Thorough proof-reading and spell checking. There are several typos here and there (e.g., "funtional" instead of "functional")
- Amend all reference to delivering "transparency" (as in "Alternatively, multidimensional outputs could be aggregated without losing transparency" or "To ensure model transparency"), as the proposed analysis does not guarantee turning neural nets from black-boxes to transparent ones. 
- Comment in the background/intro about the effort that human analysts should do, and which pipeline they could follow in inspecting a given neural network with DORA.
- Shrink Defs 6,7 while expand/better highlight in which parts they differ from Defs 3-5 (and why this matters for DORA). 
- Provide more details on the setting of experiments in Sec 5.2 and discuss more whay "human alignment" means in that context. Add a paragraph about limitations of the proposed distance metrics.


Decision: Accept (poster)