The FIX Benchmark: Extracting Features Interpretable to eXperts

## Abstract

Abstract Feature-based methods are commonly used to explain model predictions, but these methods often implicitly assume that interpretable features are readily available.
However, this is often not the case for high-dimensional data, and it can be hard even for domain experts to mathematically specify which features are important.
Can we instead automatically extract collections or groups of features that are aligned with expert knowledge? To address this gap, we present FIX (Features Interpretable to eXperts), a benchmark for measuring how well a collection of features aligns with expert knowledge.
In collaboration with domain experts, we propose FIXScore, a unified expert alignment measure applicable to diverse real-world settings across cosmology, psychology, and medicine domains in vision, language and time series data modalities.
With FIXScore, we find that popular feature-based explanation methods have poor alignment with expert-specified knowledge, highlighting the need for new methods that can better identify features interpretable to experts.

## 1 Introduction

Machine learning is increasingly used in domains like healthcare \citepDBLP:journals/corr/abs-1907-07374, law \citepATKINSON2020103387, governance \citepdoi:10.1080/01900692.2019.1575664, science \citepde_la_Torre_L_pez_2023, education \citep10.1007/978-3-319-93843-1_12 and finance \citepmodarres2018towards.
However, modern models are often black-box, which makes it hard for practitioners to understand their decision-making and safely use model outputs \citeprai2019explanation.
For example, surgeons are concerned that blind trust in model predictions will lead to poorer patient outcomes \citepHameed2023what;
in law, there are known instances of wrongful incarcerations due to over-reliance on faulty model predictions \citepZeng_2016,wexler2017computer.
Although such models have promising applications, their opaque nature is a liability in domains where transparency is crucial \citep10.1145/3442188.3445923, Hong_2020.

To address the pertinent need for transparency and explainability of their decision-making, the interpretability of machine learning models has emerged as a central focus of recent research \citeparrieta2019explainable,SAEED2023110273,räuker2023transparent.
A popular and well-studied class of interpretability methods is known as feature attributions \citepribeiro2016why,lundberg2017unified,sundararajan2017axiomatic.
Given a model and an input, a feature attribution method assigns scores to input features that reflect their respective importance toward the model’s prediction.
A key limitation, however, is that the attribution scores are only as interpretable as the underlying features themselves \citepzytek2022need.

Feature-based explanation methods commonly assume that the given features are already interpretable to the user, but this typically only holds for low-dimensional data.
With high-dimensional data like images and text documents, where the readily available features are individual pixels or tokens, feature attributions are often difficult to interpret \citepnauta2023anecdotal.
The main problem is that features at the individual pixel or token level are often too granular and thus lack clear semantic meaning in relation to the entire input.
Moreover, the important features are also domain-dependent, which means that different attributions are needed for different users.
These factors limit the usefulness of popular feature attribution methods on high-dimensional data.

Figure: Figure 1: The FIX benchmark contains 6 datasets across a diverse set of application areas, data modalities, and dataset sizes. For each dataset, we show an example of an input and some example expert features for that input.
Refer to caption: x1.png

Instead of individual features, people understand high dimensional data in terms of semantic collections of low level features, such as regions in an image or phrases in a document. Moreover, for a feature to be useful, it should align with the intuition of domain experts in the field.
To this end, an interpretable feature for high-dimensional data should have the following properties.
First, they should encompass a grouping of related low-level features (e.g., pixels, tokens), thus creating high-level features that experts can more easily digest.
Second, these low-level feature groupings should align with domain experts’ knowledge of the relevant task, thus creating features with practical relevance.
We refer to features that satisfy these criteria as expert features.

But how can we obtain such features?
In practice, it is left to domain experts to identify and provide such features for individual tasks.
Although experts often have a sense of what the expert features should be, formalizing such features is often non-trivial.
Moreover, manually annotating expert features can also be expensive and labor-intensive.
These challenges raise the critical question:

Can we automatically discover expert features that align with domain knowledge?

To this end, we present the FIX benchmark, a unified evaluation measuring feature interpretability that can capture each individual domain’s expert knowledge. Our goal is to guide the development of new methods that produce interpretable features by building a unified metric to measure how interpretable a proposed feature group is.
The FIX datasets (summarized in Figure 1) collectively encompass a diverse array of real-world settings (cosmology, psychology, and medicine) and data modalities (vision, language, and time-series signals): abdomen surgery safety identification \citepmadani2022artificial, chest X-ray classification \citeplian2021structure, mass maps regression \citepcosmogrid1, supernova classification \citeplsst, multilingual politeness classification \citephavaldar-etal-2023-comparing, and emotion classification \citepdemszky-etal-2020-goemotions, havaldar-etal-2023-multilingual.
The challenge here lies in unifying all 6 different real-world settings and 3 different data modalities into a single framework, which our proposed expert alignment measure FIXScore achieves.
This allows us to have a benchmark that does not overfit to any particular domain.
To our knowledge, while previous work has identified the need for interpretable features  \citepzytek2022need, doshivelez2017rigorous,
there does not exist yet a benchmark that measures the interpretability of features for real-world experts. The FIX benchmark accomplishes this while also serving as a basis for studying, constructing, and extracting expert features.
In summary, our contributions are as follows:

- 1.
In collaboration with domain experts, we develop the FIX benchmark, a set of 6 curated datasets with evaluation metrics for extracting Features Interpretable to eXperts in real-world settings from diverse modalities of images, text, and time-series data.
(^1^11Code and updates are available at [https://brachiolab.github.io/fix/](https://brachiolab.github.io/fix/))
- 2.
We introduce a general feature evaluation metric, FIXScore, that unifies the different real-world settings of cosmology, psychology, and medicine into a single framework. We worked closely with real domain experts to develop criteria for what made features interpretable in each domain.
- 3.
We evaluate commonly used techniques for extracting higher-level features and find that existing methods score poorly on FIXScore, highlighting the need for developing new general-purpose methods designed to automatically extract expert features.

## 2 Related Work

Interpretability.
Interpretability in machine learning is often viewed as a multifaceted concept that encompasses algorithmic transparency \citepshin2019role,rader2018explanations,grimmelikhuijsen2023explaining, explanation methods \citepmarcinkevivcs2023interpretable, havaldar2023topex, and visualization techniques \citepchoo2018visual,spinner2019explainer,wang2023deepbio, among other aspects.
In this work, we focus on feature-level interpretability, a central topic in interpretability research \citepHong_2020,nauta2023anecdotal.
Feature-based methods are popular because they are believed to offer simple, adaptable, and intuitive settings in which to analyze and develop interpretable machine learning workflows \citepmolnar2020interpretable.
We refer to \citepnauta2023anecdotal,dwivedi2023explainable,weber2023beyond and the references therein for extensive reviews on feature-based explanations.

Application-grounded Evaluation.
\citetchaleshtori2023on extend the work of \citetdoshivelez2017rigorous to propose a comprehensive taxonomy of evaluating explanations.
Notably, this includes application-grounded evaluations, which broadly seek to measure the efficacy of feature-based methods in settings with human users and realistic tasks, such as AI-assisted decision-making.
However, the available literature on application-grounded evaluations is sparse: \citetchaleshtori2023on reviewed over 50 existing NLP datasets and found that only four were suitable for application-grounded evaluations \citepdeyoung2019eraser,wadden2020fact,koreeda2021contractnli,malik2021ildc.
A principal objective of the FIX benchmark is to provide an application-grounded evaluation of feature-based explanations in real-world settings.

Feature Generation.
Because high-quality and interpretable features may not always be available, there is interest in automatically generating them by combining low-level features \citepijcai2017p352,erickson2020autogluontabular,zhang2023openfe.
Notably, \citetzhang2023openfe propose a method for tabular data using the expand-and-reduce framework \citepkanter2015deep.
However, existing generation methods do not necessarily produce interpretable features, and most works focus on tabular data.
The FIX benchmark aims to address these limitations by providing a setting in which to study and develop methods for interpretable feature generation across diverse problem domains.

XAI Benchmarks.
There exists a suite of benchmarks for explanations that cover the properties of faithfulness (or fidelity) \citepelectronics10050593, agarwal2022openxai, robustness \citepDBLP:journals/corr/abs-1806-08049, agarwal2022openxai, simulatability \citepmills2023almanacs, fairness \citepDBLP:journals/corr/abs-2112-04417, agarwal2022openxai, among others.
Quantus \citephedstrom2023quantus,
XAI-Bench \citepDBLP:journals/corr/abs-2106-12543, OpenXAI \citepagarwal2022openxai, GraphXAI \citepagarwal2023evaluating,
and ROAR \citephooker2019benchmark
are notable open-source implementations that evaluate for such properties.
CLEVR-XAI \citepARRAS202214 and \citetzhang2023xai provide benchmarks that combine vision and text.
ERASER \citepdeyoung2019eraser is a popular NLP benchmark that unifies diverse NLP datasets of human rationales and decisions.
In general, however, there is a lack of interpretability benchmarks that evaluate feature interpretability in real-world settings — a gap we aim to address with the FIX benchmark.

## 3 Expert Feature Extraction

Figure: Figure 2: The FIX benchmark allows measuring alignment of extracted features with expert features in different domains, either implicitly with a scoring function or explicitly with expert annotations.
Refer to caption: extracted/5914302/figures/images/implicit_vs_explicit_narrow.png

Feature-based explanation methods require interpretable features to be effective.
For example, surgeons communicate safety in surgery with respect to key anatomical structures and organs, which are interpretable features for surgeons \citepstrasberg2010rationale,hashimoto2019surgical.
These interpretable features are a key bridge that can help surgical AI assistants communicate effectively with surgeons.
However, ground-truth annotations for such interpretable features are often expensive and hard to obtain, as they typically require trained experts to manually annotate large amounts of data.
This bottleneck is not unique to surgery, and such challenges motivate us to study the problem of extracting features interpretable to experts, or expert features.

Consider a task with inputs from $\mathcal{X}\subseteq\mathbb{R}^{d}$ and outputs in $\mathcal{Y}$.
In the example of surgery, $\mathcal{X}$ may be the set of surgery images, and $\mathcal{Y}$ is the target of where it is safe or unsafe to operate.
We model a higher-level expert feature of input $x\in\mathcal{X}$ as a subset of features represented with a binary mask $g\in\{0,1\}^{d}$, where $g_{i}=1$ if the $i$th feature is included and 0 otherwise.
In surgery, for example, a good mask $\beta$ is one that accurately selects a key anatomical structure or organ from an input $x$.
The objective of interpretable feature extraction is to find a set of masks $\hat{G}\subseteq\{0,1\}^{d}$ that effectively approximates the expert features of $x$.
That is, each binary mask $\hat{g}\in\hat{G}$ aims to identify some subset of features meaningful to experts.

### 3.1 Measuring Alignment of Extracted Features with Expert Features

At the core of FIX is a general framework for measuring the quality of extracted features with respect to expert knowledge.
Let $\hat{G}$ be a proposed set of expert features for an input $x\in\mathbb{R}^{d}$, and suppose there exists a function $\mathrm{\textsc{ExpertAlign}}(\hat{g},x)\in[0,1]$ that captures how well a single extracted feature $\hat{g}$ is expert-interpretable for $x$.
Here, a score of $1$ means that a domain expert considers $\hat{g}$ highly interpretable, whereas a score of 0 0 means that $\hat{g}$ is a highly uninterpretable feature.
Then, given a set of proposed groups $\hat{G}$ and input $x$, we measure the quality of $\hat{G}$ for $x$ as:

$$ $\mathrm{\textsc{FIXScore}}(\hat{G},x)=\frac{1}{d}\sum_{i=1}^{d}\frac{1}{|\hat{ G}[i]|}\sum_{\hat{g}\in\hat{G}[i]}\mathrm{\textsc{ExpertAlign}}(\hat{g},x).$ (1) $$

where let $\hat{G}[i]=\{\hat{g}\in\hat{G}:i\in\hat{g}\}$ be the subset $\hat{G}$ that cover feature $i$.
Intuitively, FIXScore is an average of averages: the expert alignment for each individual feature $i=1,\ldots,d$ is averaged over all covers $\hat{G}[i]$.
This metric has two key strengths:

- 1.
Duplication Invariance at Optimality. If one extracts perfect expert features (i.e., with an alignment score of 1), the FIXScore cannot be increased further by duplicating expert features. This property ensures that the score cannot be trivially inflated with repeats.
- 2.
Encourages Diversity of Expert Features. Since the score aggregates a value for each feature from $i=1,\dots,d$, adding a new expert feature that does not yet overlap with already extracted features is always beneficial.

The use of a generic expert alignment function enables the FIXScore to accommodate a diverse set of applications. There are two main ways one can specify the ExpertAlign function: implicitly with a score specified by an expert or explicitly with annotations from an expert, as shown in Figure 2.

Case 1: Implicit Expert Alignment.
Suppose we do not have explicit annotations of expert features for ground truth groups. In this case, we use implicit expert features defined indirectly via a scoring function that measures the quality of an extracted feature. The exact formula of the score is specified by an expert and will depend on the domain and task. Implicit expert features have the advantage of potentially being more scalable than features manually annotated by experts. The Mass Maps, Supernova, Multilingual Politeness, and Emotion datasets are examples of the implicit features case.

Case 2: Explicit Expert Alignment.
In the case where we do have annotations for expert features $G^{\star}$, we can use a standardized expression for the FIXScore that measures the best possible intersection with the annotated expert features.
Then, the expert alignment score of a feature group $\hat{g}$ is

$$ $\textsc{ExpertAlign}(\hat{g},x)=\max_{g^{\star}\in G^{\star}(x)}\textsc{Match} (\hat{g},g^{\star}),\quad\text{where}\enskip\textsc{Match}(\hat{g},g^{\star})= \frac{\lvert\hat{g}\cap g^{\star}\rvert}{\lvert\hat{g}\cup g^{\star}\rvert},$ (2) $$

and $\lvert\cdot\rvert$ counts the number of ones-entries, and $\cap$ and $\cup$ are the element-wise conjunction and disjunction of two binary vectors, respectively.
In other words, Match is an intersection-over-union score.
Our notation is motivated by the fact that one can treat expert features $\hat{g}$ like sets as they are binary vectors.
The Cholecystectomy and Chest X-ray datasets have explicit expert features.

Our goal in FIX is to benchmark general-purpose feature extraction techniques that are *domain agnostic* and do not use the FIXScore during training. Instead, benchmark challengers can use neural network models trained on the end-to-end tasks to automatically extract features without explicit supervision, which we release as part of the benchmark and discuss further in Appendix B. Annotations for expert features are too expensive to collect at scale for training, while implicit features are by no means comprehensive. The FIX benchmark is intended for evaluation purposes to spur research in general purpose and automated expert feature extraction.

## 4 FIX Datasets

In this section, we briefly describe each FIX dataset in Figure 1. For each dataset, we provide an overview of the domain task and the problem setup. We then introduce the key expert alignment function that measures the quality of an expert feature, and explain why certain properties incorporated in the expert alignment function are desirable to experts.

### 4.1 Mass Maps Dataset

Motivation.
A major focus of cosmology is the initial state of the universe, which can be characterized by various cosmological parameters such as $\Omega_{m}$, which relates to energy density, and $\sigma_{8}$, which pertains to matter fluctuations \citepAbbott_2022.
These parameters influence what is observable by mass maps, also known as weak lensing maps, which capture the spatial distribution of matter density in the universe.
Although mass maps can be obtained through the precise measurement of galaxies \citepy3-massmapping,y3-shapecatalog, it is not known how to directly measure $\Omega_{m}$ and $\sigma_{8}$.
This has inspired machine learning efforts to predict the two cosmological parameters from simulations \citepribli2019weak,matilla2020weaklensing,Fluri_2022.
However, it is hard for cosmologists to gain insights into how to predict $\Omega_{m}$ and $\sigma_{8}$ from black-box ML models.

Problem Setup.
Our dataset contains clean simulations from CosmoGridV1 \citepcosmogrid1.
Each input is a one-channel image of size $(66,66)$, where the task is to predict $\Omega_{m}$ and $\sigma_{8}$.
Here, $\Omega_{m}$ is the average energy density of all matter relative to the total energy density, including radiation and dark energy, while $\sigma_{8}$ describes fluctuations in the distribution of matter \citepAbbott_2022.
The dataset has 90,000/10,000/10,000 mass maps in train/validation/test splits.

Expert Features.
When inferring $\Omega_{m}$ and $\sigma_{8}$ from the mass maps, we aim to discover which cosmological structures most influence these parameters.
Two types of cosmological structures in mass maps known to cosmologists are voids and clusters \citepmatilla2020weaklensing.
An example is illustrated in Figure 3, where voids are large regions that are under-dense relative to the mean density and appear as dark, while clusters are over-dense and appear as bright dots.

To quantify the interpretability of an expert feature in the mass maps, we develop an implicit expert alignment scoring function.
Intuitively, a group that is purely void or purely cluster is more interpretable in cosmology, while a group that is a mixture is less interpretable.
We thus develop the purity metric based on the entropy among void/cluster pixels \citepzhang2003entropy weighted by the ratio of interpretable pixels in the expert feature.
We give additional details in Appendix A.1.

$$ $\textsc{ExpertAlign}(\hat{g},x)=\mathrm{Purity}_{vc}(\hat{g},x)\cdot\mathrm{ Ratio}_{vc}(\hat{g},x)$ (3) $$

Figure: Figure 3: An example with expert features for Mass Maps Regression, showing (a) the full map, (b) a feature with 100% void, and (c) a feature with 100% cluster. Voids are under-dense large regions that appear to be dark, and clusters are over-dense regions that appear as bright dots. The purity scores for both void and cluster are 1. We gray-out the pixels not selected in each feature.
Refer to caption: x2.png

### 4.2 Supernova Dataset

Motivation.
The astronomical time-series classification, as mentioned in \citeptheplasticcteam2018photometric, involves categorizing astronomical sources that change over time.
Astronomical sources include transient phenomena (e.g., supernovae, kilonovae) and variable objects (e.g., active galactic nuclei, Mira variables).
This task analyzes simulation datasets that emulate future telescope observations from the Legacy Survey of Space and Time (LSST) \citeplsst.
Given the vastness of the universe, it is essential to identify the time periods that have the most significant impact on the classification of astronomical sources to optimize telescope observations.
Time periods with no observed data are less useful.
To avoid costly searching over all timestamps for high-influence time periods, we aim to identify significant timestamps that are linearly consistent in specific wavelengths.

Problem Setup.
We take parts of the dataset from the original PLAsTiCC challenge \citeptheplasticcteam2018photometric.
The input data are simulated LSST observations comprising four columns: observation times (modified Julian days), wavelength (filter), flux values, and flux error. The dataset encompasses 7 distinct wavelengths that work as filters, and the flux values and errors are recorded at specific time intervals for each wavelength.
The classification task is to predict whether or not each of 14 different astronomical objects exists.
The supernova dataset contains 6274/728/792 train/valid/test examples.

Expert Features.
A feature with linearly consistent flux for each wavelength is considered more interpretable in astrophysics. An illustration of expert features used for supernova classification is presented in Figure  4. This example showcases the flux value and error for various wavelengths, each represented by a different color. We colored the timestamp of expert features with the wavelength color with the highest linear consistency score. For the timestamp where there is no data point, we do not recognize it as an expert feature.
We create a linear consistency metric to assess the expert alignment score of a proposed feature in the context of a supernova.
Our linear consistency metric uses $p$, the percentage of data points that display linear consistency, penalized by $d$, the percentage of time stamps containing data points:

$$ $\displaystyle\mathrm{\textsc{ExpertAlign}}(\hat{g},x)=\max_{w\in W}p(\hat{g},x _{w})\cdot d(\hat{g},x_{w}).$ (4) $$

where $W$ is the set of unique wavelength.
Further details are provided in Appendix A.2.

Figure: Figure 4: An example with expert features for supernova classification, showing (left) the original time-series dataset and (right) an example of the interpretable expert feature group. We highlight the expert feature groups with the highest expert align scores.
Refer to caption: extracted/5914302/figures/supernova_org.png

### 4.3 Multilingual Politeness Dataset

Motivation. Different cultures express politeness differently \citepleech2007politeness, pishghadam2012study. For instance, politeness in Japan often involves acknowledging the place of others \citepspencer2016bases, whereas politeness in Spanish-speaking countries focuses on establishing mutual respect \citepplacencia2017spanish. Therefore, grounding interpretable features that indicate politeness is language-dependent. Previous work from \citetdanescu2013computational and \citetmingyang2020politelex use past politeness research to create lexica that indicate politeness/rudeness in English and Chinese, respectively.
A lexicon is a set of categories where each category contains a curated list of words.
For instance, the English politeness lexicon contains categories like Gratitude: “appreciate”, “thank you”, et cetera, and Apologizing: “sorry”, “apologies”, etc.
\citethavaldar-etal-2023-comparing expand on these theory-grounded lexica to include Spanish and Japanese.

Problem Setup. The multilingual politeness dataset from \citephavaldar-etal-2023-comparing contains 22,800 conversation snippets from Wikipedia’s editor talk pages. The dataset spans English, Spanish, Chinese, and Japanese, and native speakers of these languages have annotated each conversation snippet for politeness level, ranging from -2 (very rude) to 0 (neutral) to 2 (very polite).

Expert Features. When extracting interpretable features for a task like politeness classification across multiple languages, it is useful to ground these features using prior research from communication and psychology. If extracted politeness features from an LLM are interpretable and domain-aligned, they should match what psychologists have determined to be key politeness indicators.
Examples of expert-aligned features are shown in Table 1.
Concretely, for each lexical category, we use an LLM to embed all the contained words and then average the resulting embeddings to get a set $C$ of $k$ centroids: $C={c_{1},c_{2},...c_{k}}$. See Appendix A.3 for more details. Then, a proposed expert feature $\hat{g}\in\{0,1\}^{d}$ indicates whether or not each of the $d$ words $w_{1},w_{2},...,w_{d}\in x$ are included in the feature, and the expert alignment score for the proposed feature $\hat{g}$ can be computed as follows:

$$ $\mathrm{\textsc{ExpertAlign}}(\hat{g},x)=\max_{c\in C}\frac{1}{\lvert\hat{g} \rvert}\sum_{i=1}^{d}\hat{g}_{i}\cdot\cos(\text{embedding}(w_{i}),c)$ (5) $$

**Table 1: Examples and expert features with high expert alignment for Multilingual Politeness (top) and Emotion (bottom). These expert features correspond to low distance within the emotion circumplex and high similarity with politeness lexica, respectively.**
| Example | Expert Features with High Alignment |
| --- | --- |
| [Politeness] I was running my spellchecker and totally didn’t realize that this was a vandalized page. Please accept my apology. I will spellcheck a little slower next time. | $g_{1}$ = I, my, I |
| $g_{2}$ = spellchecker, vandalized, little, slower |  |
| $g_{3}$ = will |  |
| $g_{4}$ = my, apology |  |
| [Emotion] This was potentially the most dangerous stunt I have ever seen someone do. One minor mistake and you die. | $g_{1}$ = dangerous, die |
| $g_{2}$ = potentially, minor |  |
| $g_{3}$ = mistake, stunt |  |
| $g_{4}$ = I, someone, you |  |

### 4.4 Emotion Dataset

Motivation. Emotion classification involves inferring the emotion (e.g., Joy, Anger, etc.) reflected in a piece of text.
Researchers study emotion to build systems that can understand emotion and thus adapt accordingly when interacting with human users.
For extracted features to be useful for such systems, they must be relevant to emotion.
For example, a word like “puppy” may be used more frequently in comments labeled with Joy vs. other emotions; therefore, it may be extracted as a relevant feature for the Joy class.
However, this is a spurious correlation — emotional expression is not necessarily tied to a subject, and comments containing “puppy” may also be angry or sad.

Problem Setup.
The GoEmotions dataset from \citetdemszky-etal-2020-goemotions contains 58,000 English Reddit comments labeled for 27 emotion categories, or “neutral” if no emotion is applicable. The input is a text utterance of 1-2 sentences extracted from Reddit comments, and the output is a binary label for each of the 27 emotion categories.

Expert Features.
Example expert features are shown in Table 1.
To measure how emotion-related a feature is, we use the circumplex model of affect \citeprussell1980circumplex.
The circumplex model assumes that all emotions can be projected onto the 2D unit circle with respect to two independent dimensions –
arousal (the magnitude of intensity or activation) and valence (how negative or positive).
By projecting features onto the unit circle, we can quantify emotional relations.
In particular, we calculate the following two attributes of the features with a group: (1) their emotional signal, i.e., mean distance to the circumplex and (2) their emotional relatedness, i.e., mean pairwise distance within the circumplex.
We then calculate the following: $\textrm{Signal}(\hat{g},x)$, which measures the average Euclidean distance to the circumplex for every projected feature in $\hat{g}$, and $\textrm{Relatedness}(\hat{g},x)$, which measures the average pairwise distance between every projected feature in $\hat{g}$ (details in Appendix A.4). For an extracted feature $\hat{g}$, the expert alignment score can then be computed by:

$$ $\mathrm{\textsc{ExpertAlign}}(\hat{g},x)=\mathrm{tanh}\left(\exp\left[-\mathrm {Signal}(\hat{g},x)\cdot\mathrm{Relatedness}(\hat{g},x)\right]\right)$ (6) $$

Figure: Figure 5: An example with expert features for Chest X-Ray dataset. (a) The full X-ray image where the following pathologies are present: effusion, infiltration, and pneumothorax; (b-c) Expert-interpretable anatomical structures of the left and right lungs.
Refer to caption: extracted/5914302/figures/images/chestx_image.png

### 4.5 Chest X-Ray Dataset

Motivation.
Chest X-ray imaging is a common procedure for diagnosing conditions such as atelectasis, cardiomegaly, and effusion, among others.
Although radiologists are skilled at analyzing such images, modern machine learning models are increasingly competitive in diagnostic performance \citepahmad2021reviewing.
Therefore, ML models may prove useful in assisting radiologists in making diagnoses.
However, in the absence of an explanation, radiologists may only trust the model output if it matches their own predictions.
Moreover, inaccurate AI assistants are shown to negatively affect diagnostic performance \citepyu2024heterogeneity.
To address this problem, explainability could be employed as a safeguard to help radiologists decide whether or not to trust the model.
As such, it is important for machine learning models to provide explanations for their diagnoses.

Problem Setup.
We use the NIH-Google dataset \citepmajkowska2020chest available from the TorchXRayVision library \citepcohen2022xrv.
This is a relabeling of the NIH ChestX-ray14 dataset \citepwang2017chestx which improved the quality of the original labels.
It contains 28,868 chest X-ray images labeled for 14 common pathology categories: atelectasis, calcification, cardiomegaly, etc.
We randomly partition the dataset into train/test splits of 23,094 and 5,774, respectively.
The task is a multi-label classification problem for identifying the presence of each pathology.

Expert Features.
Radiology reports commonly refer to anatomical structures (e.g., spine, lungs), which allows radiologists to perform and communicate accurate diagnoses to patients.
We provide these expert-interpretable features in the form of anatomical structure segmentations.
However, because we could not find datasets with both pathology labels and anatomical segmentations, we used a pre-trained model from TorchXRayVision to generate the structure labelings for each image.
We use explicit expert alignment as described in Equation 2 to compute alignment of an extracted feature $\hat{g}$ and the 14 predicted anatomical structure segments, including the left clavicle, heart, etc.
Details of the Chest X-Ray dataset can be found in Appendix A.5.

Figure: Figure 6: An example with expert features of Laparoscopic Cholecystectomy Surgery Dataset: (a) The view of the surgeon sees; (b) The safe region for operations; (c) The gallbladder, a key anatomical structure for the critical view of safety.
Refer to caption: extracted/5914302/figures/images/cholec_image.png

### 4.6 Laparoscopic Cholecystectomy Surgery Dataset

Motivation.
Laparoscopic cholecystectomy (gallbladder removal) is one of the most common elective abdominal surgeries performed in the US, with over 750,000 operations annually \citepstinton2012epidemiology.
A common complication of laparoscopic surgery is bile duct injury, which is associated with an 8-fold increase in mortality \citepmichael2020safe and accounts for more than $1B in US healthcare annual spending \citepberci2013laparoscopic.
Notably, 97% of such complications result from human visualization errors \citepway2003causes.
The surgery site commonly contains obstructing tissues, inflammation, and other patient-specific artifacts — all of which may prevent the surgeon from getting a perfect view.
Consequently, there is growing interest in harnessing advanced vision models to help surgeons distinguish safe and risky areas for operation.
However, experienced surgeons rarely trust model outputs due to their opaque nature, while inexperienced surgeons might overly rely on model predictions.
Therefore, any safe and useful machine learning model must be able to provide explanations that align with surgeons’ expectations.

Problem Setup.
The task is to identify the safe and unsafe regions for incision.
We use the open-source subset of the data from \citepmadani2022artificial, wherein the authors enlist surgeons to annotate surgery video data from the
M2CAI16 workflow challenge \citepstauder2016tum and Cholec80 \citeptwinanda2016endonet datasets.
This dataset consists of 1015 annotated images with a random train/test split of 812 and 203, respectively.

Expert Features.
In cholecystectomy, it is a common practice for surgeons to identify the *critical view of safety* before performing any irreversible operations \citepstrasberg2010rationale,hashimoto2019surgical.
This view identifies the location of vital organs and structures that inform the safe region of operation and is incidentally what surgeons often need as part of an explanation.
We provide these expert-interpretable labels in the form of organ segmentations (liver, gallbladder, hepatocystic triangle).
We use explicit expert alignment as described in Equation 2 to compute alignment of an extracted feature $\hat{g}$ and the surgeon-annotated organ labels taken from \citetmadani2022artificial.
Details of the Cholecystectomy dataset can be found in Appendix A.6.

## 5 Baseline Algorithms & Discussion

We evaluate standard techniques widely used within the vision, text, and time series domains to create higher-level features.
We provide a brief summary below, with additional details in Appendix C.

**Table 2: Baselines scores of different FIX settings. We report the mean score and give a more comprehensive table in Appendix C. We describe baseline implementations in Section 5. One thing to note is that FIXScore is not comparable for different tasks (e.g. between Mass Maps and Supernova) as the data and specific expert alignment metrics are different for different tasks.**
|  | Vision | Time Series | Language |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | Method | Cholec | ChestX | MassMaps | Method | Supernova | Method | Politeness | Emotion |
| Domain-specific | Identity | 0.4686 | 0.2154 | 0.5483 | Identity | 0.0152 | Identity | 0.6070 | 0.0103 |
| Random | 0.1086 | 0.0427 | 0.5505 | Random | 0.0358 | Random | 0.6478 | 0.0303 |  |
| Patch | 0.0323 | 0.0999 | 0.5555 | Slice 5 | 0.0337 | Words | 0.6851 | 0.1182 |  |
| Quickshift | 0.2622 | 0.3419 | 0.5492 | Slice 10 | 0.0555 | Phrases | 0.6351 | 0.0198 |  |
| Watershed | 0.2807 | 0.1452 | 0.5590 | Slice 15 | 0.0554 | Sentences | 0.6109 | 0.0120 |  |
| SAM | 0.3678 | 0.3151 | 0.5521 |  |  |  |  |  |  |
| CRAFT | 0.0271 | 0.1175 | 0.3996 |  |  |  |  |  |  |
| Domain-agnostic | Clustering | 0.2880 | 0.2627 | 0.5515 | Clustering | 0.2622 | Clustering | 0.6680 | 0.0912 |
| Archipelago | 0.3351 | 0.2148 | 0.5542 | Archipelago | 0.2574 | Archipelago | 0.6773 | 0.0527 |  |

Domain-specific Baselines.
We consider the following domain-centric baselines.
(Image)
For image data, we consider three segmentation methods \citepkim2024gof.
Patches \citepdosovitskiy2021image divides the image into grids where each cell is the same size.
Quickshift \citep1704833 connects similar neighboring pixels into a common superpixel.
Watershed \citep4154796 simulates flooding on a topographic surface.
CRAFT \citepfel2023craft generates concept attribution maps.
(Time-series)
For time series data, we take equal size slices of the data across time as patches \citepschlegel2021ts. We use different slice sizes to see how they impact multiple baselines.
We take various slice sizes, such as 5, 10, and 15, separately to evaluate the results of multiple baselines.
(Text)
For text data, we present three baselines for extracting features \citeprychener2022granularity.
At the finest granularity, we treat each word as a feature.
The second baseline considers each phrase as a feature. Phrases are comprised of groups of words that are separated by some punctuation in the original text.
At the coarsest granularity, we treat each sentence as a feature.

Domain-agnostic Baselines.
We additionally consider the following domain-agnostic baselines for feature extraction.
(Identity) We combine all elements into one single group.
(Random)
We select features at random, up to the maximum baseline results for the group. The group maximum is calculated as: $(\text{group maximum})\approx(\text{scaling factor})\times(\text{number of
expert features})$.
The size of the distinct expert feature varies depending on the setting, and further details for each setting can be found in Appendix C. We use a scaling factor of about 1.5 to allow for flexibility.
(Clustering)
For images, we first use Quickshift to generate segments and then pass each segment through a feature extractor (ResNet-18 by default).
For time series, we use raw features from each time segment.
We then apply K-means clustering on the extracted/raw features to relabel and merge segments.
For text, we use BERTopic \citepgrootendorst2022bertopic to obtain the clusters.
(Archipelago)
We adapt the implementation of Archipelago \citeptsang2020how to use ResNet-18 with quickshift for feature extraction.

Results and Discussions.
We show results on the baselines in Table 2.
For image datasets, Quickshift has the best performance compared to Patch and Watershed on both the Cholecystectomy dataset and the Chest X-ray dataset, since they have natural images.
All baselines perform similarly for the Mass Maps dataset.
That the range of mass maps is different from other tasks is potentially because they are not natural images, but rather similar to topographic surfaces.
For the Supernova time-series dataset, larger slices score yield higher expert alignment scores.
For both Multilingual Politeness and Emotion datasets, individual words appear to be the most expert-aligned features. Generally, however, we see that the domain-agnostic neural baselines tend to also perform better than or close to the best domain-centric baseline. The main benefit of using a neural approach is that it can more easily automatically discover relevant features.

## 6 Conclusion

We propose FIX, a curated benchmark of datasets with evaluation metrics for extracting expert features in diverse real-world settings.
Our benchmark addresses a gap in the literature by providing researchers with an environment to study and automatically extract interpretable features for experts.

Limitations and Future Work.
The FIX benchmark is not an exhaustive specification of all expert features, and may fail to capture others types. The ones we included are generally non-controversial and well-accepted by the domain’s expert community, but we can foresee that there are cases where this may not be true. Dealing with potential conflicting expert opinions may need a more nuanced approach, which is left for future work to address.
Furthermore, although we cover cosmology, psychology, and medicine domains in this work, the metrics for these domains may not be appropriate for all settings. We encourage prospective users to consider and implement metrics most appropriate to their particular settings.
Future work includes the development of new, general purpose techniques that can extract expert features from data and models without supervision.

## Appendix A Dataset Details

All datasets and their respective Croissant metadata records and licenses are available on HuggingFace at the following links.

- •
Mass Maps:
[https://huggingface.co/datasets/BrachioLab/massmaps-cosmogrid-100k](https://huggingface.co/datasets/BrachioLab/massmaps-cosmogrid-100k)
- •
Supernova:
[https://huggingface.co/datasets/BrachioLab/supernova-timeseries](https://huggingface.co/datasets/BrachioLab/supernova-timeseries)
- •
Multilingual Politeness:
[https://huggingface.co/datasets/BrachioLab/multilingual_politeness](https://huggingface.co/datasets/BrachioLab/multilingual_politeness)
- •
Emotion:
[https://huggingface.co/datasets/BrachioLab/emotion](https://huggingface.co/datasets/BrachioLab/emotion)
- •
Chest X-Ray:
[https://huggingface.co/datasets/BrachioLab/chestx](https://huggingface.co/datasets/BrachioLab/chestx)
- •
Laparoscopic Cholecystectomy Surgery:
[https://huggingface.co/datasets/BrachioLab/cholecystectomy](https://huggingface.co/datasets/BrachioLab/cholecystectomy)

### A.1 Mass Maps Dataset

#### Problem Setup.

We randomly split the data to consist of 90,000 train and 10,000 validation maps and maintain the original 10,000 test maps.
We follow the post-processing procedure in \citety3-massmapping,you2023sumofparts for low-noise maps.
Following previous works \citepribli2019weak,matilla2020weaklensing,Fluri_2022,you2023sumofparts, we use a CNN-based model for predicting $\Omega_{m}$ and $\sigma_{8}$.

#### Metric.

Let $x\in\mathbb{R}^{d}$ be the input mass map with $d=H\times W$ pixels, and $g\in\{0,1\}^{d}$ be a boolean mask $g$ that describes which pixels belong to the group, where $g_{i}=1$ if the $i$th pixel belongs to the group, and 0 otherwise.

We can compute the purity score of each group to void and cluster.
We say a pixel is a void (underdensed) pixel if its intensity is below 0 0, and a cluster (overdensed) pixel if its intensity is above $3\sigma(x)$, following previous works \citepmatilla2020weaklensing,you2023sumofparts.
We first compute the proportion of void pixels and cluster pixels in feature $g$

$$ $\displaystyle P_{v}(g,x)=\frac{\sum_{i=1}^{d}\mathbbm{1}[g_{i}x_{i}<0]}{g^{ \intercal}\mathbf{1}},\quad\quad P_{c}(g,x)=\frac{\sum_{i=1}^{d}\mathbbm{1}[g_ {i}x_{i}>3\sigma(x)]}{g^{\intercal}\mathbf{1}}$ (7) $$

where $\mathbf{1}\in 1^{d}$ is the identity matrix, the numerators count the number of underdensed or overdensed pixels, and $g^{\intercal}\mathbf{1}$ is the number of pixels in the feature.
In practice, we add a small $\epsilon=10^{-6}$ to $P_{v}$ and $P_{c}$ and renormalize them, to avoid taking the log of 0 later.
Next, we compute the proportion of pixels that are void or cluster, only among the void/cluster pixels:

$$ $\displaystyle P_{v}^{\prime}(g,x)=\frac{P_{v}(g,x)}{P_{v}(g,x)+P_{c}(g,x)}, \quad\quad P_{c}^{\prime}(g,x)=\frac{P_{c}(g,x)}{P_{v}(g,x)+P_{c}(g,x)}$ (8) $$

Then, we compute the ExpertAlign score for the predicted feature $\hat{g}$ by computing the void/cluster-only entropy reversed and scaled to $[0,1]$, weighted by the percentage of void/cluster pixels among all pixels.

$$ $\displaystyle\mathrm{Purity}_{vc}(\hat{g},x)=$ $\displaystyle\frac{1}{2}(2+P_{v}^{\prime}(\hat{g},x)\log_{2}P_{v}^{\prime}( \hat{g},x)+P_{c}^{\prime}(\hat{g},x)\log_{2}P_{c}^{\prime}(\hat{g},x))$ (9) $$

where $-(P_{v}^{\prime}(\hat{g},x)\log_{2}P_{v}^{\prime}(\hat{g},x)+P_{c}^{\prime}(
\hat{g},x)\log_{2}P_{c}^{\prime}(\hat{g},x))$ is the entropy computed only on void and cluster pixels, a close to 0 score indicating that the interpretable portion of the feature is mostly void or cluster.
$\mathrm{Purity}_{vc}(\hat{g},x)$ is 0 if among the pixels in the proposed feature that are either void or cluster pixels, half are void and half are cluster pixels, and 1 if all are void or all are cluster pixels, regardless of how many other pixels there are in the proposed feature.

We also have the ratio

$$ $\displaystyle\mathrm{Ratio}_{vc}(\hat{g},x)=(P_{v}(\hat{g},x)+P_{c}(\hat{g},x))$ (10) $$

which is the total proportion of the feature that is any interpretable feature type at all.

We then have our ExpertAlign for Mass Maps:

$$ $\textsc{ExpertAlign}(\hat{g},x)=\mathrm{Purity}(\hat{g},x)\cdot\mathrm{Ratio}( \hat{g},x)$ (11) $$

which is then 0 when all the pixels in the feature are neither void or cluster, and 1 if all pixels are void pixels or all pixels are cluster pixels, and somewhere in the middle if most pixels are void or cluster pixels but there is a mix between both.

### A.2 Supernova Dataset

Problem Setup.
We extracted data from the PLAsTiCC Astronomical Classification challenge \citeptheplasticcteam2018photometric. (^2^22[https://www.kaggle.com/c/PLAsTiCC-2018](https://www.kaggle.com/c/PLAsTiCC-2018)) PLAsTiCC dataset was designed to replicate a selection of observed objects with type information typically used to train a machine learning classifier. The challenge aims to categorize a realistic simulation of all LSST observations that are dimmer and more distorted than those in the training set. The dataset contains 15 classes, with 14 of them present in the training sample. The remaining class is intended to encompass intriguing objects that are theorized to exist but have not yet been observed.

In our dataset, we split the original training set into 90/10 training/validation, and the original test set was uploaded unchanged. We made these sets balanced for each class. The class includes objects such as tidal disruption event (TDE), peculiar type Ia supernova (SNIax), type Ibc supernova (SNIbc), and kilonova (KN).
The dataset contains four columns: observation times (modified Julian days, MJD), wavelength (filter), flux values, and flux error. Spectroscopy measures the flux with respect to wavelength, similar to using a prism to split light into different colors.

Due to the expected high volume of data from upcoming sky surveys, it is not possible to obtain spectroscopic observations for every object. However, these observations are crucial for us. Therefore, we use an approach to capture images of objects through different filters, where each filter selects light within a specific broad wavelength range. The supernova dataset includes 7 different wavelengths that are used. The flux values and errors are recorded at specific time intervals for each wavelength. These values are utilized to predict the class that this data should be classified into.

#### Metric.

We use the following expert alignment metric to measure if a group of features is interpretable:

$$ $\displaystyle\mathrm{\textsc{ExpertAlign}}(\hat{g},x)=\max_{w\in W}\mathrm{ LinearConsistency}(\hat{g},x_{w})$ (12) $$

where $W$ is the set of unique wavelength, $\hat{g}$ is the feature group, and $x_{w}$ is the subset of $x$ within wavelength $w$.
In the supernova setting, there are three parameters: $\epsilon$, the parameter for how much standard deviation $\sigma$ is allowed, window size $\lambda$ and the step size $\tau$.
Therefore, we formulate the $\mathrm{LinearConsistency}$ function as follows:

$$ $\displaystyle\@add@centering\mathrm{\centering LinearConsistency}(\hat{g},x_{w })=p(\hat{g},x_{w})\cdot d(\hat{g},x_{w})$ (13) $$

$p(\hat{g},x_{w})$ is the percentage of data points that display linear consistency, penalized by $d(\hat{g},x_{w})$, which is the percentage of time steps containing data points.

Let $\beta(x,y)=\operatorname*{arg\,min}_{\beta}(X^{T}\beta-y)^{2}$, where $X=\begin{bmatrix}x&1\end{bmatrix}$ and $\beta=\begin{bmatrix}\beta_{1}&\beta_{0}\end{bmatrix}$. Here, $\beta_{1}$ is the slope and $\beta_{0}$ is the intercept. $M$ is the number of data points in $x_{w}$, and $\hat{y}_{w,i}={x}_{w,i}\cdot\beta$. Then, we have

$$ $\displaystyle\mathrm{p}(\hat{g},x_{w})=\frac{1}{M}\sum_{i=1}^{M}\mathbbm{1}[ \hat{y}_{w,i}\in[{y}_{w,i}-\epsilon\cdot\omega_{w,i},{y}_{w,i}+\epsilon\cdot \omega_{w,i}]]$ (14) $$

Let $t_{1},...,t_{N}$ be time steps at step size intervals. Then $t_{i}=t_{start}+i*\tau$, and $N$ is the number of time steps. We also have

$$ $\displaystyle\mathrm{d}(\hat{g},x_{w})=\frac{1}{N}\sum_{i=1}^{N}\mathbbm{1}[ \exists_{i}:{x}_{w,i}\in[{t}_{i},{t}_{i}+\lambda]]$ (15) $$

A higher $\mathrm{\textsc{ExpertAlign}}(\hat{g},x)\in[0,1]$ value means the flux slope at each wavelength is consistently linear and there are not many time intervals without data.

### A.3 Multilingual Politeness Dataset

#### Problem Setup.

This politeness dataset from \citethavaldar-etal-2023-multilingual is intended for politeness classification, and would likely be solved via a fine-tuned multilingual LLM. Namely, this would be a regression task, using a trained LLM to output the politeness level of a given conversation snippet as a real number ranging from -2 to 2.

The dataset is accompanied by a theory-grounded politeness lexica. Such lexica built with domain expert input have been promising for explaining style \citepdanescu2013computational, culture \citephavaldar-etal-2024-building, and other such complex multilingual constructs.

#### Metric.

Assume a theory-grounded Lexica $L$ with $k$ categories: $L={\ell_{1},\ell_{2},...\ell_{k}}$, where each set $\ell_{i}\subseteq\mathcal{W}$, where $\mathcal{W}$ is the set of all words. For each category, we use an LLM to embed all the contained words and then average the resulting embeddings, to get a set $C$ of $k$ centroids: $C={c_{1},c_{2},...c_{k}}$. We define this formally as:

$$ $C:\left\{\frac{1}{|\ell_{i}|}\sum_{w\in l_{i}}\text{embedding}(w)\text{ for all }i\in[1,k]\right\}$ (16) $$

For a group $\hat{g}$ containing words $w_{1},w_{2},...$, the group-level expert alignment score can be computed as follows:

$$ $\mathrm{\textsc{ExpertAlign}}(\hat{g},x)=\max_{c\in C}\frac{1}{|\hat{g}|}\sum_ {w\in\hat{g}}\cos(\text{embedding}(w),c)$ (17) $$

Note that each language has a different theory-grounded lexicon, so we calculate a unique domain alignment score for each language.

### A.4 Emotion Dataset

#### Problem Setup.

This dataset is intended for emotion classification and is currently solved with a fine-tuned LLM \citepdemszky-etal-2020-goemotions. Namely, this is a classification task where an LLM is trained to select some subset of 28 emotions (including neutrality) given a 1-2 sentence Reddit comment.

#### Projection onto the Circumplex.

**Table 3: Emotions used to define the valence and arousal axis anchors for projection into the Valence-Arousal plane. We select the 5 emotions from the circumplex closest to each axis point.**
| Axis Anchor | Russell Emotions |
| --- | --- |
| Positive valence (PV) | Happy, Pleased, Delighted, Excited, Satisfied |
| Negative valence (NV) | Miserable, Frustrated, Sad, Depressed, Afraid |
| High arousal (HA) | Astonished, Alarmed, Angry, Afraid, Excited |
| Low arousal (LA) | Tired, Sleepy, Calm, Satisfied, Depressed |

To define the valence and arousal axes, we first generate four axis-defining points by averaging the contextualized embeddings ("I feel [emotion]") of the emotions listed in Table 3. This gives us four vectors in embedding space – positive valence ($\vec{v}_{\mathrm{pos}}$), negative valence($\vec{v}_{\mathrm{neg}}$), high arousal($\vec{a}_{\mathrm{high}}$), and low arousal($\vec{a}_{\mathrm{low}}$). We mathematically describe our projection function below:

- 1.
We define the valence axis, $V$, as $\vec{v}_{\mathrm{pos}}-\vec{v}_{\mathrm{neg}}$ and the arousal axis, $A$, as $\vec{a}_{\mathrm{high}}-\vec{a}_{\mathrm{low}}$. We then normalize $V$ and $A$ and calculate the origin as the midpoints of these axes: $(\vec{v}_{\mathrm{middle}},\vec{a}_{\mathrm{middle}})$.
- 2.
We then scale the axes so $\vec{v}_{\mathrm{pos}}$, $\vec{v}_{\mathrm{neg}}$, $\vec{a}_{\mathrm{high}}$, and $\vec{a}_{\mathrm{low}}$ anchor to $(1,0)$, $(-1,0)$, $(0,1)$, and $(0,-1)$ respectively. This enforces the circumplex to be a unit circle in the valence-arousal plane.
- 3.
We compute the angle $\theta$ between the valence-arousal axes by solving $\cos\theta=\frac{{V}\cdot{A}}{\lVert V\rVert\cdot\lVert A\rVert}$
- 4.
For each embedding vector $\vec{x}$ in the set $\{x_{i}\}_{i=1}^{n}$ we want to project into our defined plane, we compute the valence and arousal components for $x_{i}$ as follows:
$x_{i}^{v}=(x_{i}-\vec{v}_{\mathrm{middle}})\cdot\vec{V}$
$x_{i}^{a}=(x_{i}-\vec{a}_{\mathrm{middle}})\cdot\vec{A}$.
- 5.
We calculate the x and y coordinates to plot, enforcing orthogonality between the axes:
$\tilde{x_{i}^{v}}=x_{i}^{v}-x_{i}^{a}\cdot\cos\theta$
$\tilde{x_{i}^{a}}=x_{i}^{a}-x_{i}^{v}\cdot\cos\theta$
- 6.
Finally, we plot $(\tilde{x_{i}^{v}},\tilde{x_{i}^{v}})$ in the Valence-Arousal plane. We then calculate the shortest distance from $(\tilde{x_{i}^{v}},\tilde{x_{i}^{v}})$ to the circumplex unit circle.

Figure: Figure 7: The circumplex model of affect [russell1980circumplex].
Refer to caption: extracted/5914302/figures/circumplex.png

#### Metric.

We calculate the following two values for a proposed feature $\hat{g}$ containing words $w_{1},w_{2},...$, where $n$ is the number of words in $\hat{g}$:

$$ $\displaystyle\mathrm{Signal}(\hat{g})$ $\displaystyle=\frac{1}{n}\sum_{w\in\hat{g}}\left|\left\|\mathrm{Proj}(w)\right \|_{2}-1\right|$ (18) $\displaystyle\mathrm{Relatedness}(\hat{g})$ $\displaystyle=\frac{1}{n^{2}}\sum_{i}^{n}\sum_{j}^{n}\|\mathrm{Proj}(w_{i})- \mathrm{Proj}(w_{j})\|_{2}$ (19) $$

where $\mathrm{Signal}(\hat{g},x)$ measures the average Euclidean distance to the circumplex for every projected feature in $\hat{g}$, and $\mathrm{Relatedness}(\hat{g},x)$ measures the average pairwise distance between every projected feature in $\hat{g}$. We formalize the expert alignment metric as follows.
For a group $\hat{g}$, the expert alignment score can be computed by:

$$ $\mathrm{\textsc{ExpertAlign}}(\hat{g},x)=\mathrm{tanh}\left(\exp\left[-\mathrm {Signal}(\hat{g},x)\cdot\mathrm{Relatedness}(\hat{g},x)\right]\right)$ (20) $$

### A.5 Chest X-Ray Dataset

We used datasets and pretrained models from TorchXRayVision \citepcohen2022xrv.(^3^33[https://github.com/mlmed/torchxrayvision](https://github.com/mlmed/torchxrayvision))
In particular, we use the NIH-Google dataset \citepmajkowska2020chest, which is a relabeling of the NIH ChestX-ray14 dataset \citepwang2017chestx.
This dataset contains 28,868 chest X-ray images labeled for 14 common pathology categories, with a train/test split of 23,094 and 5,774.
We additionally used a pre-trained structure segmentation model to produce 14 segmentations.
The task is a multi-label classification problem for identifying the presence of each pathology.
The 14 pathologies are:

> Atelectasis, Cardiomegaly, Consolidation, Edema, Effusion, Emphysema, Fibrosis, Hernia, Infiltration, Mass, Nodule, Pleural Thickening, Pneumonia, Pneumothorax

The 14 anatomical structures are:

> Left Clavicle, Right Clavicle, Left Scapula, Right Scapula, Left Lung, Right Lung, Left Hilus Pulmonis, Right Hilus Pulmonis, Heart, Aorta, Facies Diaphragmatica, Mediastinum, Weasand, Spine

### A.6 Laparoscopic Cholecystectomy Surgery Dataset

We use the open-source subset of the data from \citepmadani2022artificial, which consists of surgeon-annotated video data taken from the
M2CAI16 workflow challenge \citepstauder2016tum and Cholec80 \citeptwinanda2016endonet datasets.
The task is to identify the safe/unsafe regions of where to operate.
Specifically, each pixel of the image has one of three labels: background, safe, or unsafe.
The expert labels provide each pixel with one of four labels: background, liver, gallbladder, and hepatocystic triangle.

## Appendix B Interpretable Feature Extraction Details

Figure 8 illustrates a graphical model representing the Interpretable Feature Extraction pipeline for a given FIX dataset.

Figure: Figure 8: We illustrate a graphical model representing the Interpretable Feature Extraction pipeline for a given FIX dataset, with FIXScore metric in its general form. There are $m$ true feature groups $g$ and $m$ latent features $\ell$, and $m^{\prime}$ proposed feature groups $\hat{g}$ and $m^{\prime}$ proposed latent features $\hat{\ell}$. $m$ does not have to equal $m^{\prime}$. Moreover, $n$ indicates the number of examples in the dataset. The person figure on near the closest arrow indicates that a domain expert would be able to infer the variable on the right-hand side of the arrow from the variable on the left-hand side arrow. In addition, $\epsilon$ is included to account for noise.
Refer to caption: x5.png

## Appendix C Baselines Details

The FIX benchmark is publicly available at: [https://brachiolab.github.io/fix/](https://brachiolab.github.io/fix/)

#### Bootstrapping.

For each setting’s baselines experiments, we use a bootstrapping method (with replacement) to estimate the standard deviation of the sample means of FIXScore.

#### Group Maximum.

For the number of groups, we take the scaling factor multiplied by the size of the distinct expert feature, which differs for each setting. The scaling factor we choose across all setting is 1.5 (and round up to the next nice whole number).

In the case of a supernova setting, we consider a distinct expert feature size of 6. This is because the maximum number of distinct expert features we can obtain is 6, given that there are a maximum of 3 humps in the time series dataset. For each hump, there are both peaks and troughs, leading to a potential maximum of 6 distinct expert features.

For the multilingual politeness setting, the group maximum would be 40, which is the total number of lexical categories, 26, with the scaling factor multiplied in to give some flexibility.

For the emotion setting, the group maximum would be , which is the total number of lexical categories, 26, with the scaling factor multiplied in to give some flexibility.

For mass maps, the group maximum would be 25. We compute the maximum number of local maximums 7 on mass maps blurred with $\sigma=3$ and local minimums 7 on mass maps blurred with $\sigma=5$, which sums up to be 14. We can then multiply with the scaling factor to give some flexibility and then we round up to 25.

#### Baseline Parameters.

For mass maps, we use the following parameters for baselines. For patch, we use $8\times 8$ grid. For QuickShift, we use kernel size 5, max dist 10, and sigma 0.2. For watershed, we use min dist 10, compactness 0. For SAM, we use ‘vit_h’. For Archipelago, we use the same Quickshift parameters for the Quickshift segmenter.

#### Baseline Results.

We report the full baseline results with standard deviations in Table 4.

**Table 4: Baselines of different FIX settings. We report the mean FIXScore for all examples in each setting, with standard deviations.**
|  | Method | Cholecystectomy | Chest X-ray | Mass Maps |
| --- | --- | --- | --- | --- |
| Image | Identity | 0.4686 $\pm$ 0.0096 | 0.2154 $\pm$ 0.0027 | 0.5483 $\pm$ 0.0015 |
| Random | 0.1086 $\pm$ 0.0004 | 0.0427 $\pm$ 0.0001 | 0.5505 $\pm$ 0.0014 |  |
| Patch | 0.0323 $\pm$ 0.0001 | 0.0999 $\pm$ 0.0008 | 0.5555 $\pm$ 0.0013 |  |
| Quickshift | 0.2622 $\pm$ 0.0034 | 0.3419 $\pm$ 0.0025 | 0.5492 $\pm$ 0.0009 |  |
| Watershed | 0.2807 $\pm$ 0.0051 | 0.1452 $\pm$ 0.0017 | 0.5590 $\pm$ 0.0017 |  |
| SAM | 0.3678 $\pm$ 0.0074 | 0.3151 $\pm$ 0.0064 | 0.5521 $\pm$ 0.0009 |  |
| CRAFT | 0.0271 $\pm$ 0.0007 | 0.1175 $\pm$ 0.0011 | 0.3996 $\pm$ 0.0023 |  |
| Domain-Agnostic | Clustering | 0.2880 $\pm$ 0.0049 | 0.2627 $\pm$ 0.0039 | 0.5515 $\pm$ 0.0014 |
| Archipelago | 0.3351 $\pm$ 0.0034 | 0.2148 $\pm$ 0.0009 | 0.5542 $\pm$ 0.0014 |  |
|  |  | Supernova |  |  |
| Time Series | Identity | 0.0152 $\pm$ 0.0011 |  |  |
| Random | 0.0358 $\pm$ 0.0021 |  |  |  |
| Slice 5 | 0.0337 $\pm$ 0.0015 |  |  |  |
| Slice 10 | 0.0555 $\pm$ 0.0044 |  |  |  |
| Slice 15 | 0.0554 $\pm$ 0.0032 |  |  |  |
| Domain-Agnostic | Clustering | 0.2622 $\pm$ 0.0037 |  |  |
| Archipelago | 0.2574 $\pm$ 0.0082 |  |  |  |
|  |  | Multilingual Politeness | Emotion |  |
| Text | Identity | 0.6070 $\pm$ 0.0015 | 0.0103 $\pm$ 0.0001 |  |
| Random | 0.6478 $\pm$ 0.0012 | 0.0303 $\pm$ 0.0004 |  |  |
| Words | 0.6851 $\pm$ 0.0010 | 0.1182 $\pm$ 0.0003 |  |  |
| Phrases | 0.6351 $\pm$ 0.0010 | 0.0198 $\pm$ 0.0003 |  |  |
| Sentences | 0.6109 $\pm$ 0.0006 | 0.0120 $\pm$ 0.0002 |  |  |
| Domain-Agnostic | Clustering | 0.6680 $\pm$ 0.0048 | 0.0912 $\pm$ 0.0005 |  |
| Archipelago | 0.6773 $\pm$ 0.0006 | 0.0527 $\pm$ 0.0008 |  |  |

## Appendix D Compute Resources

All experiments were conducted on two server machines, each with 8 NVIDIA A100 GPUs and 8 NVIDIA A6000 GPUs, respectively.

## Appendix E Safeguards

The datasets and models that we use in this work are not high risk and are previously open-source and publicly available. In particular, for our medical settings which would pose the most potential safety concern, the datasets we sourced our FIX datasets from are already open-source and consists of de-anonymized images.

## Appendix F Datasheets

We follow the documentation framework provided by \citetgebru2021datasheets to create datasheets for the FIX datasets. We address each section per dataset.

### F.1 Motivation

#### For what purpose was the dataset created?

- •
Mass Maps: The original dataset, CosmoGridV1 \citepcosmogrid1, was created to help with predicting the initial states of the universe in cosmology.
- •
Supernova: The original dataset PLAsTiCC for Kaggle competition \citepallam2018photometric, was created to classify astronomical sources that vary with time into different classes.
- •
Multilingual Politeness: The Multilingual Politeness dataset \citephavaldar-etal-2023-comparing was created to holistically explore how politeness varies across different languages.
- •
Emotion: The original dataset, GoEmotions \citepdemszky-etal-2020-goemotions, was created to help understand emotion expressed in language.
- •
Chest X-Ray: The NIH-Google dataset \citepmajkowska2020chest, which is a relabeling of the NIH ChestX-ray14 dataset \citepwang2017chestx, was created to help identify the presence of common pathologies.
- •
Laparoscopic Cholecystectomy Surgery: The original datasets from M2CAI16 workflow challenge \citepstauder2016tum and Cholec80 \citeptwinanda2016endonet were created to help identify the safe and unsafe areas of surgery.

#### Who created the dataset (e.g., which team, research group) and on behalf of which entity (e.g., company, institution, organization)?

- •
Mass Maps: The original dataset CosmoGridV1 \citepcosmogrid1 was created by Janis Fluri, Tomasz Kacprzak, Aurel Schneider, Alexandre Refregier, and Joachim Stadel at the ETH Zurich and the University of Zurich.
The simulations were run at the Swiss Supercomputing Center (CSCS) as part of the project “Measuring Dark Energy with Deep Learning”, hosted at ETH Zurich by the IT Services Group of the Department of Physics.
We adapt the dataset and add a validation split.
- •
Supernova: The original dataset PLAsTiCC was created by \citettheplasticcteam2018photometric. We adapt the dataset, add a validation split, and balance the sets for each class.
- •
Multilingual Politeness: The Multilingual Politeness dataset \citephavaldar-etal-2023-comparing was created by Shreya Havaldar, Matthew Pressimone, Eric Wong, and Lyle Ungar at the University of Pennsylvania.
- •
Emotion: The original GoEmotions \citepdemszky-etal-2020-goemotions dataset was created by Dorottya Demszky, Dana Movshovitz-Attias, Jeongwoo Ko, Alan Cowen, Gaurav Nemade, and Sujith Ravi at Stanford University, Google Research and Amazon Alexa.
- •
Chest X-Ray: The NIH-Google dataset \citepmajkowska2020chest was created by Anna Majkowska, Sid Mittal, David F Steiner, Joshua J Reicher, Scott Mayer McKinney, Gavin E Duggan, Krish Eswaran, Po-Hsuan Cameron Chen, Yun Liu, Sreenivasa Raju Kalidindi, et al., at Google Health, Stanford Healthcare and Palo Alto Veterans Affairs, Apollo Radiology International, and California Advanced Imaging.
- •
Laparoscopic Cholecystectomy Surgery: The M2CA116 workflow challenge dataset \citepstauder2016tum was created by Ralf Stauder, Daniel Ostler, Michael Kranzfelder, Sebastian Koller, Hubertus Feußner, and Nassir Navab at Technische Universität München in Germany and Johns Hopkins University. The Cholec80 dataset \citeptwinanda2016endonet was created by Andru P Twinanda, Sherif Shehata, Didier Mutter, Jacques Marescaux, Michel De Mathelin, and Nicolas Padoy, at ICube, University of Strasbourg, CNRS, IHU, University Hospital of Strasbourg, IRCAD and IHU Strasbourg, France.

#### Who funded the creation of the dataset?

- •
Please refer to each setting’s respective papers for funding details.

### F.2 Composition

- •
The answers are described in our paper. Please refer to Section 4 and Appendix A for more details.

### F.3 Collection Process

- •
We defer the collection process to the relevant works that created them. Please refer to Section 4 and Appendix A for more details.

### F.4 Preprocessing/cleaning/labeling

- •
The answers are described in our paper. Please refer to Section 4 and Appendix A for more details.

### F.5 Uses

- •
The answers are described in our paper. Please refer to Section 4 and Appendix A for more details.

### F.6 Distribution

#### Will the dataset be distributed to third parties outside of the entity (e.g., company, institution,
organization) on behalf of which the dataset was created?

- •
No. Our datasets will be managed and maintained by our research group.

#### How will the dataset will be distributed (e.g., tarball on website, API, GitHub)?

- •
The FIX datasets are released to the public and hosted on Huggingface (please refer to links in Appendix A).

#### When will the dataset be distributed?

- •
The datasets have been released now, in 2024.

#### Will the dataset be distributed under a copyright or other intellectual property (IP) license, and/or under applicable terms of use (ToU)?

- •
Mass Maps: The Mass Maps dataset is distributed under CC BY 4.0, following the original dataset CosmoGridV1 \citepcosmogrid1.
- •
Supernova: The Supernova dataset is distributed under the MIT license.
- •
Multilingual Politeness: The Multilingual Politeness dataset is distributed under the CC-BY-NC license.
- •
Emotion: The Emotion dataset is distributed under the Apache 2.0 license.
- •
Chest X-Ray: The Chest X-Ray dataset is distributed under the Apache 2.0 license.
- •
Laparoscopic Cholecystectomy Surgery: The Laparoscopic Cholecystectomy Surgery dataset is distributed under the CC by NC SA 4.0 license.

## Appendix G Author Statement

We bear all responsibility for any potential violation of rights, etc., and confirmation of data licenses.

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 5
presentation: 2
questions: - The FIXScore relies on predefined expert alignment metrics, which may be subjective and vary across experts within a domain. Would different expert definitions for the same task lead to huge variations in evaluating the extracted features?
- Table 2 shows that the results on Domain-specific Time Series are much lower than others, but the domain agnostic is rather comparable to vision tasks. And Emotion score is also lower than Politeness. While the caption notes that FIXScore is not comparable across tasks, is there any interpretation of these score discrepancies? Additionally, the abstract concludes “poor alignment with expert-specified knowledge.” If scores are not directly comparable across datasets, how should we interpret a score as "high" or "low" and determine alignment quality?
soundness: 2
strengths: - The paper presents a novel benchmark to measure the alignment of machine-extracted features with expert knowledge, which is valuable for guiding future research on general-purpose, automated expert feature extraction.
- The paper employs datasets from various domains, including tabular, text, and vision data, including tabular, text and vision data, and evaluates various feature extraction methods, including both domain-specific and domain-agnostic techniques.
summary: The paper presents "FIX", a benchmark designed to evaluate how well machine learning-generated features align with expert knowledge across various domains like cosmology, psychology, and medicine. The authors introduce FIXScore, a unified measure of feature interpretability, providing a framework for assessing alignment with expert knowledge in vision, language, and time-series data. Through empirical analysis, the authors highlights the limitations of current feature extraction methods and underscores the need for methods that can produce expert-interpretable features, particularly for high-dimensional data.
weaknesses: - Although the paper uses five datasets for experimentation, it lacks a clear rationale for selecting these specific datasets.
- The structure of the paper could be improved - the results and discussions of the results are very brief in the main paper and lack of interpretability. The paper could consider condense the dataset introduction part and add more discussions on the results to better demonstrate the insights.
- There is an inconsistency in line 478, where the paper mentions “three segmentation methods” for image data but lists four: “Patches,” “Quickshift,” “Watershed,” and “CRAFT.” Additionally, Table 2 includes an extra method, "SAM," which is not cited.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: see above
soundness: 2
strengths: - the problem the paper tries to address is important. 
- the datasets cover different domains
summary: This paper proposes a benchmark for measuring how well a collection of features aligns with expert knowledge.
weaknesses: - The formulation of the ExpertAlign metric for various datasets appears somewhat ad hoc, lacking clear justification for the chosen metrics. Why were these particular metrics selected over other potential designs, and how do they relate to Formula 2?
- If the proposed metrics were part of a broader methodological paper as a way to evaluate the primary method, they would likely be suitable. However, as a standalone contribution, they seem insufficient.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: Please refer to the weakness section for open questions regarding the work.
soundness: 3
strengths: 1. The proposed framework provides an important benchmark encompassing six diverse real-world settings and three different data modalities and unifies them using the introduced expert alignment measure FixScore.

2. The authors provide a very detailed implementation detail of how they calculate the scores for each dataset in the Appendix.

3. The authors raise an important problem of grounding feature attribution in explainable artificial intelligence to knowledge from domain experts.
summary: Explainable Artificial Intelligence (XAI) has been an active research direction to understand the black-box nature of highly complex deep neural networks. One of the widely researched areas in XAI is feature attribution, where we attribute the prediction of a given model back to its input features, i.e., assigning scores to input features that reflect their respective importance toward the model’s prediction. The authors argue the usefulness of feature attribution techniques is limited for high-dimensional data, where the identified important features do not have any semantic meaning and are uninterpretable to stakeholders and domain experts. To address the above challenges, the authors define **expert features** as low-level feature groupings that align with domain experts’ knowledge of the relevant task and have practical relevance. Further, they present the FIX benchmark, a unified evaluation measuring feature interpretability that captures the knowledge of domain experts from diverse applications and unifies them using the FixScore metric, which quantifies how interpretable
a proposed feature group is.
weaknesses: 1. *"We propose FIXScore, a unified expert alignment measure With FIXScore, we find that popular feature-based explanation methods have poor alignment with expert-specified knowledge, highlighting the need for new methods to better identify features interpretable to experts."* -- Are the authors proposing a call to the community to develop new feature attribution explanation methods? If yes, how does FixScore or expert features help in this regard? Most classification models are trained on certain downstream objectives which do not necessarily reward the model to learn the task using expert features. Assuming complete faithfulness of explanation methods, aren't we arguing that the underlying training paradigm should change?

2. The authors argue that *"the main problem is that features at the individual pixel or token level are often too granular and thus lack clear semantic meaning in relation to the entire input"* -- While I agree that individual pixels and tokens do not possess any clear semantic meaning, the model was never trained to learn features that align with domain experts, i.e., they are always trained on a given cross-entropy or some other downstream loss function which may/may not align with domain experts. Hence, this argument is not necessarily a symptom of the feature attribution methods but could be implied from the underlying training of the model itself.

3. The authors propose some specific properties of grouping low-level features to high-level features that are more digestible and the low-level features should align with domain experts’ knowledge of the relevant task -- Wouldn't training a model using concept bottleneck models suffice these properties?

4. The authors argue about the importance of feature attribution that is grounded in domain expert knowledge. However, in sec 4.1, the authors use the purity- and ratio-based metrics that are proxies to ground truth information needed to align attributed features. Since these purity and ratio metrics are essentially based on pixel intensities, isn't this another measure of something like the Intersection of Union (IoU), which we use to evaluate feature attribution maps for real-world images and calculate the IoU between the generated importance map with ground-truth object bounding boxes?

5. For datasets like emotion and politeness, getting expert features may be highly subjective. Are there ways the authors are thinking to address this problem? Further, these subjective notions have different levels to it and are mostly context and scenario-dependent. How do these factors affect the underlying expert features?

6. One way to support the significance of the expert features would have been to train a machine learning model on just the features that are deemed to be aligned with domain experts and report the accuracy of the trained model. If the expert features are indeed grounded in domain knowledge, a model should be able to achieve high predictive performance using only those features.

**(Minor points)**

1. It would be great for the reader if the authors could include the metric calculation of each of the datasets in the main paper as that is one of the main contributions of the paper.

2. The anonymous link to the codes doesn't work.

---

confidence: 2
contribution: 2
initial_rating: 3
presentation: 3
questions: 1. What are the performances of the models trained for the given task? How difficult is it to train such models on these datasets? Are the performances sufficient to draw any conclusions about the quality of explanations or could these problems also be inherently incorporated in the trained models?
2. How do you ensure that the trained models actually align with the expert features in their reasoning?
3. What requirements should models satisfy to be eligible for an evaluation of explanations using these datasets? Would you include the models as part of the benchmark?
soundness: 2
strengths: 1. The evaluation of feature-based explanations with respect to interpretability is an important topic, where human-annotated ground-truth explanations specifically tailored to a given task and domain are an important tool to evaluate approaches.
2. The proposed datasets cover multiple domains, and the proposal of a single score for each task is good for comparing methods. The applications seem interesting, although I cannot judge on the quality of any of the proposed expert features.
3. The paper is well-structured and easy to follow, and the purpose of the paper is clear
summary: This paper proposes a real-word benchmark to evaluate feature-based explanations of black box machine learning models. The authors collect six real-word datasets from three different domains to evaluate the performance of feature attribution methods in terms of pre-defined "expert features" $G^*$, i.e. a collection of feature sets that provide a meaningful interpretable "ground-truth" explanation. This is in contrast to existing work that focuses mainly on general metrics and synthetic examples without including domain knowledge specifically tailored to the given dataset and task. Given an instance $x$ and a collection of feature sets $\hat G$, i.e. the proposed explanation, the FIX benchmark computes essentially a heuristical score that determines how well $\hat G$ and $G^*$ align: For each feature $i$, the average "amount of alignment" of all elements in $\hat G$ that involve feature $i$ is compared with the ground-truth elements of $G^*$. This alignment is computed using the "ExpertAlign" function, which is either task-specific or simply the maximum intersection over union metric with the human-annotated ground truth "expert features". This score is then averaged over all features $i$ (Eq. 1). The benchmark covers the extraction of "expert features" for all six datasets, where for two datasets human-annotated are used, and the remaining four datasets have "implicit expert features", which are defined indirectly via a task-specific scoring functions. The paper then proceeds to evaluate several domain-specific pre-processing techniques (such as patches, watershed for images) and some feature-based explanations (such as Archipelago, CRAFT) to evaluate their score on the benchmark. From these results the authors conclude that current feature-based explanations are not sufficiently aligned with "expert features".
weaknesses: 1. **Lack of popular feature attribution methods:** The paper proposes a benchmark that has the potential to improve explanations based on groups of features. However, the benchmarking of methods is significantly under-developed. The paper claims (e.g. in the abstract) that feature-based explanations fall short in identifying the given "expert features". This claim is not sufficiently covered by the experiments. The authors do barely evaluate feature-based explanations. In the experiments, from current XAI literature only Archipelago is considered (and image-specific variants, such as CRAFT). While in the introduction feature attributions, such as LIME or SHAP are discussed, none of these concepts appear in the empirical evaluation. In contrast, the evaluation in Table 2 centers around pre-processing techniques rather than feature attribution methods. For improvement, to support this claim, I would suggest to include a variety of well-established feature attribution methods, such as SHAP, Integrated Gradients, LIME and their variants for higher-order explanations and evaluate their performance on the FIX benchmark. Moreover, it is unclear to me, why the proposed FIXScores reported in Table 2 are insufficient / unsatisfactory. 
2. **Unclear modeling choices:** Given only the datasets, it is difficult to execute a benchmarking of XAI methods, since a black box model is required. The paper does not address the question on how to model these datasets in the first place. A potential improvement would be to clearly discuss the modeling aspect of the datasets, or incorporate the trained model in the benchmark. As of now, it is unclear if any results obtained by the benchmark is due to a bad modeling choice or a bad interpretability method. Moreover, it is even unclear if the "expert features" and actual model reasoning are aligned, which would be crucial for the benchmark (see Q1-Q3 below).
3. **Implications remain unclear:** The paper in its current state has limited impact on future research, since it does not provide clear evidence of limitations in current XAI methods (see above), nor where these are rooted in or how they could be mitigated in the future. It would be very helpful to clearly evaluate how these methods fail, why, and to propose improvements or directions for future developments.