Detecting Systematic Weaknesses in Vision Models along Predefined Human-Understandable Dimensions
Authors: Sujan Sai Gannamaneni
Rohil Prakash Rao
Michael Mock
Maram Akila
Stefan Wrobel
Venue: Accepted by TMLR ()

Abstract: Slice discovery methods (SDMs) are prominent algorithms for finding systematic weaknesses in DNNs. They identify top-k semantically coherent slices/subsets of data where a DNN-under-test has low performance. For being directly useful, slices should be aligned with human-understandable and relevant dimensions, which, for example, are defined by safety and domain experts as part of the operational design domain (ODD). While SDMs can be applied effectively on structured data, their application on image data is complicated by the lack of semantic metadata. To address these issues, we present an algorithm that combines foundation models for zero-shot image classification to generate semantic metadata with methods for combinatorial search to find systematic weaknesses in images. In contrast to existing approaches, ours identifies weak slices that are in line with predefined human-understandable dimensions. As the algorithm includes foundation models, its intermediate and final results may not always be exact. Therefore, we include an approach to address the impact of noisy metadata. We validate our algorithm on both synthetic and real-world datasets, demonstrating its ability to recover human-understandable systematic weaknesses. Furthermore, using our approach, we identify systematic weaknesses of multiple pre-trained and publicly available state-of-the-art computer vision DNNs.

Published in Transactions on Machine Learning Research (07/2025)
Detecting Systematic Weaknesses in Vision Models along Predefined Human-Understandable Dimensions
Sujan Sai Gannamaneni Prounhofer TAIS, Lamorr Instätute
Rohil Prakash Rao Praunhofer IAIS
Michael Mock Prounhofer TAIS
Maram Akila Praunhofer IAIS, Lamarr Institute
Stefan Wrobel
Prounhofer TATS, University of Bonn
Reviewed on OpenReview: https: openreviee.. net/ forum? id= 9K9put4nBB
Abstract
Slice discovery methods (SDMs) are prominent algorithms for finding systematic weaknesses in DNNs. They identify top-k semantically coherent slices/subsets of data where a DNN- under-test has low performance. For being directly useful, slices should be aligned with Humaa--understnndable and relevant dimensions, which, for example, are defined by safety and domain experts as part of the operational design domain (ODD). While SDMs can be applied effectively on structured data, their application on image data is complicated by the lack of semantic metadata. To address these issues, we present an algorithm that combines foundation models for zero-shot image classification generate semantie metadata with methods for combinatorial search to find systematic weaknesses in images. In contrast to existing approaches, ours identifies weak slices that are in line with predefined human- understandable dimensions. As the algorithm includes foundation models, its intermediate and final results may not always be exact. Therefore, we include an approach to address the impact of noisy metadata. We validate our algorithm on both synthetic and real-world ability recover human-understandabll systematic weaknesses. Furthermore, using our approach, we identify systematic weaknesses of multiple pre-trained and publicly available state-of-the-art computer vision DNNs.
datasots, demonstrating
1 Introduction
With recent advances in machine learning (ML), there has been significant improvement in the modeling of unstructured data, such images. However, for safety-critical applications, ML models need to be developed with a focus on trustworthiness by investigating and correcting potential failure modes. To that end, systematic errors of DNNs need to be studied and rectified. Hidden stratification Rayyden-Rayner
ltt al. 2020) and fairness-related bias [Bollamwiii & Gebru 2018 Wang et al. 2020 Li et (d.33) due to spurious correlations (Xiao
2021 Geirhos et al. 2020 Mahmood al. 2021 and underrepresented
subpopulations (Sagawa* al. 2020 Santurkar al. 2021 are some examples of potential failure modes where the weakness systematic in nature.
The existence of these modes implies that there

---

Published in Transactions on Machine Learning Research (07/2025)
slices of data where the performance of the DNN-under-test (DuT) is worse than the average performance on the entire test dataset. Although identifying slices with weak performance would be trivial by simply grouping samples on which models have high error, identifying slices that are both semantically coherent and have high error is challenging. This is due to the lack of semantic metadata that describes the slices for many data domains (e.g., images, text). Despite this challenge, identifying such slices provides a human- understandable global explanation of the model behavior. Morcover, semantically coherent weak slices offer actionable insights for debugging and auditing models.
From safety and certification perspective, upcoming standards (e.g., ISO/PAS 8800 ISO 2024 and works with focus on AI in automotive (Koopman & Fratrik 2019 Burton et al. 2022), acrospace Euro- pean Union Aviation Safety Agency, 2024] and railway (Zeller et al. 2023) domains have highlighted the importance of data completeness and quality using, in most cases, Operational Design Domains (ODDs). In automotive, Herrmann et al. (2022) have proposed ontologies for different traffic participants that can be used build ODDs for automated driving. The goal of using such ODDs is to describe the scope of AT applications in terms of humar-understandable,,, salety-relevant dimensions where comprehensible safely argumentations can be built w.r.t. robustness, explainability, and interpretability. To facilitate building such safety augmentations, testing approaches for ML developers and safety experts that evaluate DNN performance and identify systematic weaknesses are essential.
Although in recent years, several works (Chung et al. 2019
Sagadeeva & Boehm 2021 d Eon et al. 2022
2023 Chen et al., Eyuboglu et al.] 2022 Metzen et al.) 2023; Plumb et al. 2023 Jain et al.) 2023 Gao et al.
2023) have proposed methods for analyzing systematic weaknesses, there a lack of focus on identifying weak- nesses of models evaluated on real-world datasets, where the weaknesses align with human-understanddblle semantic concepts defined, for example, by safety experts in ODDs. We argue that it is more beneficial from safety perspective if the approaches to identify systematic weaknesses are ODD-compliant for two main reasons: (i) the slices are useful as the identified vulnerabilities are aligned with aiman-understandal safety-relevant dimensions, and (ii) the slices are actionable as ML developers can gather more data to re- train or reweight existing samples to improve performance along the safely-relevant dimensions. We address the challenge of analyzing unstructured image data by designing an algorithm that leverages recent advances in foundation models and systematic weakness analysis methods for structured data. Our contributions be summarized follows:
We introduce an algorithm that takes in an image dataset, ODD description, and performance values of a DuT as inputs and outputs systematic weaknesses of the DuT (see section 3 Our implemen- tation is available at /tttps:// www.s/aa.oooo/aaaaaaa.aaaaaaaaaa sai- wataaayyyyyettee-elllleeeeeeeeee
Concretely, part of the metadata generation module, we make use of CLIP (Radford et al. 2021 to leverage its rich joint image, lext embedding space. As part of the slice discovery module, we propose using SliceLine (Sagadeeva & Boehm 2021 with modifications to identify weak slices that align with the ODD (see section 3).
In addition, we address the noisy nature of metadata generation and propose a way to recover relevant weak slices even if CLIP labeling is suboptimal. We empirically evaluate the behavior of our algorithm at various levels of label quality using synthetic data (see section
Furthermore, we evaluate multiple pre-trained and publicly available DuTs using algorithm real-world datasets and provide insights into their systematic weaknesses (see section 5
2 Related Work
In this section, we review the recent progress in analyzing systematic weaknesses using slice discovery methods (SDMs) (Eyuboglu et al., 2022) for structured and unstructured data and highlight their connection to interpretability and fcature attribution methods.
In the literature, slices are often also called suhgroups or subsets of data. All three terms are used interchangeably in relation to systematic weakness analysis.
2

---

Published in Transactions on Machine Learning Research (07/2025)
For structured data, methods such as SliceFinder (Chung al. 2019) and SliceLine (Saaadevaa & Boehm 2021) leverage the rich metadata available in the form of features to slice the data and exhaustively search for top-k low-performing slices. The differences between these two approaches lie in the scoring of errors, the pruming strategy, and how they handle slice sizes. Although these two approaches were explicitly developed to identify systematic weaknesses, subgroup-discovvery techniques (Atzmueller 2015), a subset of data mining, have a similar problem formulation and could also potentially be used for slice discovery of structured data.
For unstructured data such as images, where metadata is not directly available, SOTA approaches have taken two lines of research. In the first line of prior work, referred to as slice-and-tag approaches by (Chen et al, 2025), for a given test dataset, DNN embeddings are used as proxies for coherency. Weak-performing slices of the data are obtained by clustering these embeddings along with model errors. Here, approaches such as Spotlight (d'Eon et al.; 2022) perform clustering on the embeddings of the final layers of the DuT itself. In contrast, recent approaches leverage the joint embedding space of foundation models such as CLIP (Radford] ct al., 2021) and apply Gaussian mixture models like in DOMINO (Eyuboglu et al. 2022 or SVMs like in SVM-FD (Jain et. al., 2023) to identify coherent clusters. In Spollight, an additional step involving humans is required to inspect and understand what uniquely constitutes weak slice. DOMINO and SVM-FD automate the slice description process to reduce human effort and bias using an additional DNN. In all these approaches, as coherence is only loosely enforced based on DNN embeddings, it is not always clear what specific human--nderstanddble concept uniquely constitutes a slice. Without, this knowledge, it would be unclear to the ML developers what new data samples would need to be collected to retrain the model and fix the systematic weakness. To mitigate this problem, some approaches (Gao et al.] 2023 Slyman et 2023 propose iterative human-in-the-llop testing to ensure that the identified slices are human-understanddllee
In the second line of prior work, referred to as tag-and-slice approaches by (Chen et al. 2025 inspired by counterfactuals and leveraging CLIP, several approaches (Wiles et al.) 2022 Metzen et al. 2023) propose syuthetically generating new (counterfactual) images that would lead to erroneous predictions by controlling the content and data shift in the image. Among these, PromptAttack (Metzen ct al., 2023) also proposes to identify weaknesses that are aligned with the ODDs. However, while PromptAttack generales new samples using image-genent mnage-generation DNNs, which could potentially introduce biases due to domain shift, our approach is more closely aligned with carlier methods that evaluate a DNN on given test dataset. In this direction, HiBug (Chen et al. 2023) utilizes GPT-based model to assign attributes to given dataset. Building this and appearing concurrently with our work, HiBug2 (Chen et al.) 2025) extends HiBug with a search algorithm to identify weak slices. While we also apply attributes to the data to perform a subsequent weak slice search, we, instead, opt for the less compute-intensive CLIP (Radford et al.) 2021 model to generate attributes. Additionally, we develop a Bayesian framework to compensate for the label noise that occurs from the attribution.
In contrast to SDMs, local interpretability and feature attribution methods (Ribeiro et al., 2016
Lundberg
& Lee 2017p, while linking achieved understandability to actionability (Guidotti et al. 2022 identify
local explanations and not the global systematic weaknesses. In addition, the feature attribution methods themselves might not always be robust or consistent (Krishna et al. 2024).
3 Method
In this section, we present our algorithm for weakness detection on the basis of human-understee understandalle se- mantic dimensions. To this end, we introduce notation regarding metadata and slicing, discuss the generation of metadata, formulate DNN weakness within a Bayesian framework to account for the impact of noise, and lastly detail how such impact can be acknowledged within slice discovery algorithms.
Consider -NN-under-test (DuT) M trained some computer vision task. Let D be the (test) dataset containing the inputs and the corresponding task-related ground truth. For cach sample E D. using some per-sample performance metric (e.g,, intersection over union (IoU) and, if applicable, applying some threshold, we obtain binarized DuT errors, defined as e E {0, Each sample is either correctly (c, = 0) or incorrectly (e, = 1) predicted by the DuT. Here, we slightly deviate from conventional notation by considering individual samples rather than the DuT inputs. Although identical for image classification, in the case of object detection, multiple samples (i.e., objects) may be present in a given input image, over
Notation:
3

---

Published in Transactions on Machine Learning Research (07 (07/2025)
Unstructured to Structured Data
Inputs
Weak Slice Discovery
Sample-bases based
Per-Object Performance
Metadata Quality Check
Binarized Errors
Samples
DuT (M)
$1
E1
C2
Systematic Weakness Detector
Metadata Generation Module
Dataset
Foundation Models
Sensors (e.g., Depth Sensors)
(D)
SWD-1:
SL + observed error
(e.g.. CLIIP)
SWD-2: SL + corrected error
SWD-3: Combined
Noisy to Accurate
Results
ODD (Z)
Dim.
Attr.
D Gender
Age
Slice
Slice Statistics
Gender
Male Female
Desc.
Age
Adult Young
$1
$2
Female
Female
Young
Adult
Age = Adult
S |1 S2 ....
Gender Female
Figure 1: Our algorithm for finding systematic weaknesses of CV models. Given a model, test dataset, and an ODD description for the objects we are interested in, we build a database of object-level performance and metadata in structured format. Weak slice discovery methods are then applied to this database to identify top-k weak slices of the model.
which inference is performed. Using set of samples with individual errors allows us to define slices S C D of the data and their corresponding (average) error rate æ defined as [S] EEes és. One of the goals of slice than the global average ēbz- However,
discovery methods is to find slices where E|s is significantly this constraint alone could be trivially satisfied by selecting all samples where e; 1. But, this, in general, would reveal no further information than the known data-points with bad performance.
As outlined in section 2 cxisting works in slice discovery can be broadly categorized into two areas of research: (i) slice-and-lag and (ii) lag-and-slice, as introduced by (Chen et al.) 2025). The first category includes methods such as DOMINO (Fyuboglu et al. 2022), Spotlight. dd'Fon et al. 2022), and SVM- FD (Jain et al. 2023 These approaches begin by embedding each input using some DNN & (e.g., CLIP the DuT itself). In the resulting embedding space, clustering classification approaches (c.g., using GMMs or SVMs) are applied to the joint space of DuT performance e and the embeddings E(x) across the full test dataset. (Plumb et al.) 2023) offers helpful overview of the different embedding and clustering approaches used in slice discovery. This constitutes the slicing step with the expectation that the obtained slices C D retain semantic coherency and also contain high DuT errors. Next, as part of the tagging step, some approaches use language models to describe S. As the approaches do not explicitly enforce the descriptions to follow set of semantics within Z, the descriptions might be very general. Moreover, there is no guarantee that coherent region in the latent space corresponds to single, lumann-nndestanndale concept. Instead, mixtures across broader categories may arise, and elements of an assumed semantic concept for S may also be reflected in data points beyond the slice. In contrast, the tag-and-slice approaches like ours and HiBug2 (Chen el. al.) 2025) focus on "tagging" each input & from a predefined set. Z of semaantic dimensions and corresponding attributes. To enable this, some DNN G (e.g., CLIP or GPT) is used to both embed and classify cach input based on the dimensions dimensions can be fairtess-related, such as "gender" or "age", with attributes like "young" or "old", but may also include other salety-relevant aspects such "occlusion" or "clothing-color". This way, unstructured
For instance, pedestrian detection, such
4

---

Published in Transactions on Machine Learning Research (07/2025)
image data is transformed into structured metadata containing the relevant semantic dimensions. As part of the slicing step, this metadata is combined with the DuT errors e and analysed using a search algorithm. These approaches offer direct interpretability of the slices, and the choice of dimensions can be aligned, e.g., with existing safety considerations. More concretely, larger parts of Operational Design Domains can often be formalized in terms of simple sets Z. For our work, the concrete set used is inspired by the ODD within Herrmann et al. (022)) from the autonomous driving domain and we use 2 and ODD synonymously in the remainder P PromptAttack (Metzen et al., 2023) also falls within the tag-and-slice category. However, in this method, instead of metadata generation for existing data, new data points s' are generated that are not within DD using diffusion models based on semantics defined in E. By using the generated s' as a test set, weaknesses of DuT are evaluated. Here, issues related to syn2real gap and systematic weaknesses of diffusion models must be considered when evaluating the results.
In fig. D we present our algorithm, where, using inputs such as a dataset D, a predefined ODD 2, and a DuT M, we transform the task of finding systematic weaknesses in the unstructured data domain into a problem in the structured data domain. The algorithm is designed with modular components for adaptability. The first module handles the generation of structured metadata, while the second module applies the weak slice search algorithm to the generated structured metadata. With a structured description of the data, we can formulate slices as rules over Z, e.g., gender = male A occlusion = (0.9, 1.0]. This allows for a more probabilistic notation p(elS) of the true error given the slice. Slice discovery is then the task of finding (coherent) conditions S such that the conditional expectation is maximized.
While there is great interest from safety experts and certification bodies in using ODDs for safety argumentation, metadata that align with the ODD are scarcely available for most, particu- larly image, domains. Human annotation of such metadata is often out of scope for large datasets due to cost and time constraints. However, an automated metadata generation module that captures different semantic dimensions of Z is feasible with existing technologies. For example, multi-modal foundation model like CLIP with its joint image and text embedding space could be a potential candidate for such automated annotation out of the box or after fine-tuning. Inspired by Gannamaneni et al. (2023), we leverage CLIP to generate metadata for each sample s in our dataset across all dimensions in Z using the embedding and classification function G. Specifically, we first obtain the embodding of an input sample s using the CLIP vision encoder. Simultancously, we generate embeddings for attribute-specific prompts using the CLIP text encoder, following the ensemble prompting strategy introduced by Gannamaneni et al. (2023), which builds upon the ensemble prompting technique proposed in Radford et al. (2021). For example, in the case of the gender dimension with attributes such as "male" and "female", we manually construct ensemble prompts for each attribute and encode them using the text. encoder. Autribute classification for each dimension is performed by computing similarity scores between the image and text embeddings; the attribute with the highest score is assigned. For example, if sample s depicts a 'male," " the correspouding similarity score would be highest, and 'male" would be assigned as the metadata value for the gender dimension for that sample. This procedure facilitates the extraction of structured metadata from unstructured image data.
Metadata Generation:
Taking the ontology for pedestrians from the automotive domain as a baseline, a qualitative evaluation of CLIP's capability was performed by GGnnamaneni et al. (2023). While CLIP achieved SOTA level zero- shot performance on different dimensions such as gender, skin-color, and age for portrait shots of human faces in the CclobA (Liu et al., 2015) datasot, they observed a drop in performance on real-world datasets containing pedestrians like in the Cityscapes (Cordts al.. 2016) dataset. The drop in performance can be attributed to more challenging conditions, such as complex poses, low illumination, and high occlusion. These obscrvations, along with our experiments, show that the function & is subject to varying degrees and types of uncertainty, depending on the dimensions of Z: (i) the presence of based (aleatoric) uncertainties, i.e., where the image resolution is low or the object in question is heavily occluded or distant, leading to errors in the generated metadata. (ii) the presence of model-based (epistemic) uncertainties, iee, where the function G exhibits suboptimal performance. While (i) can occur in the case of both human and CLIP-based
PIn practice ODDs are often ontological in nature, and only sub-components of them might be captured by scmantic sets of the form Z. For example, "trucks" and "pedestrians" could be part of the same ODD for DuT but would have different. semantic dimensions.
5

---

Published in Transactions on Machine Learning Research (07/2025)
annotation, (ii) occurs more prominently in non-human, automated labeling X Therefore, any method that aims to consider metadata generated using such techniques should take into account the incurred noise in downstream tasks.
Bayesian Framework to Account for the Impact of Noise: To address the uncertain nature of classification, we extend the previous slice notation of the error to the joint probability p(e,C,)) C,S), where C' represents the outcome of automated labeling for some attribute of a dimension, while S denotes the corresponding ground truth. For simplicity, we drop the indices and make the additional assumption that S, C can be seen as binary, ie. they may either be true (S, C) not true (-S, -C), respectively (for details on the non-binary case, see appendix B.4). Using Bayes' Theorem and marginalizing over C or S, we can express
p(e|S) -c0s)),(eeC,,,p + ple)) p(e|C) =p(elc, pleec, (1-Pe).
(1)
(2)
Here, p(e]S) represents the true slice error, while p(e|C) denotes the observed slice error. Furthermore, Pc p(S|C) and re = S(C are shorthand for precision and recall of the labeling function G measured towards the ground truth, and are used in our algorithm, fig. | for the quality check. A detailed derivation of the equations is provided in appendix B.1. Making these relations explicit allows us to investigate the hypothesis typically underlying Slice Discovery Methods in more detail. Specifically, based solely on the observed slice performance/ /weakness p(e(C), one may conclude that related data property S represents a weakness of the DuT, i.e., we assume that p(e[S) also has a comparable performance/ weekkness. weeakesssseeeaeessss While in our algorithm the relation between S andd C is explicit
the latter is given by classifier for the former, in other approaches (d'Fon et al. 2022 Eyuboglu et al. 2022 Jain et al. 2023) the relation is implicit, as observed sets C are interpreted to indicate meaning of S (typically referred to as slice label). Another assumption typically made is the independence between the labeling function G and DuT. This independence would imply that the errors of the DuT do not depend on the noise (errors) of G. Specifically, for a semantic attribute, the error rates p(e(C, when 9 is correct and the error rate p(e)-C,,S) when it not should be (approximately) equal. However, our experiments indicate that this is not always the case; therefore, we denote the difference by
8p(e|8) -
(3)
Please note that ōp describes intra-set variances of the error rate in the set and is not a conditional probability on its own. Taking into account this potential dependence, we can derive the true error from the observed error exactly given the performance of the annotation process using
KS
K-5
p(e|S)= plelC) P-C - p(cl-C) (pc - 1) Sp(e|S)
Pc + P-C -1 rc +op(el- S) (pc - 1 1)p-c
PCP-C
(4)
PC P-C -1
PC + P-C
Independence ass sesumption
correction terms
As long as the independence assumption is (approximately) valid, implying óp(e]S) x 8p(c)-S) Rs O, the slice error given the semantic attribute S is obtained by separating the two types of observed error probabilities p(elC), which possible as long as the denominator is non-zero An analysis of properties of this equation w.r.t. the denominator allows to automatically create quality indicators the validity or invalidity of the obtained corrected slices for attribute S. The full derivation and further details on quality indicators can be found in appendix B.2.
Weak Slice discovery on Structured ODD Data with SliceLine: We have now established hods to generate metadata and correct for noise during the motadata generation. With this background, in al- gorithm D we propose three stages for Systematic Weakness Detection (SWD-1,2,3). In SWD-1, using the generated structured metadata and observed errors p(e[C), we employ SliceLine (Sagadeeva & Boehm 2021
"High-quaiity human labeling typically requires multiple measures to reduce inter-observer variability or epistemic uncer- tainty in general (c.g. via labeling guides). However, in this work, we consider human labcling as high-quality compared to nase- labeling.
4 For the sake of numeric stability, also denominators which are only approximately zero should he discarded.
6

---

Published in Transactions on Machine Learning Research (07/)025)
to provide a ranked list of top-k worst performing slices based on scoring function that takes into account the errors and sizes of the slices (see eq. (5) in appendix A.4 for details on how SliceLine works). As we have motivated, observed errors may not always provide a sufficient signal to identify the underlying error (see the top row in fig. 2 Therefore, in SWD-2, using eq. 4 to compensate for noise in the metadata, we provide cor- rected errors instead of observed errors to SiiceLine to provide a second ranked list of top-k worst-performing slices S. However, as it requires extensive human effort to identify certain parameters, i.c., óp(e[S), 6p(el-S) in eq. ( in particular for combinations of semantics, we make cheaper approximation only considering the independence assumption part of the equation. This is implemented in compute@orrectterrooo () in algorithm 1 To operationalize this part of the equation, we estimate precision values based on human evaluation of metadata quality on only n = 60 samples per attribute (see appendix B.4). The subsequent corrected errors from this independence assumption are used in the SliceLine scoring function. Based on the slice quality indicators discussed above, we are also able to discard invalid slices due to denominator values close to zero. In addition to SWD-1 and SWD-2, we also consider morge of the resulting slices from SWD-1 and SWD-2, this might provide a complementary effect. We refer to this merged list as the output of SWD-3. The merge step includes sorting based the score of the slice from the scoring function, removal of duplicate slices, and filtering of invalid slices. The SliceLine hyperparameters include the level (maximal search depth), i.c. the maximal number of semantic dimensions considered simultancously, as well as cut-off for the necessary slice error øls to consider S a valid slice.
Algorithm 1: Systematic Weakness Detector (SWD)
Input: Metadata {Cz, (Cz,,C2,,.... Cz,....J, J, errors e, Precision vectors {pc), SliceLine hyper- typee-parameters Output: Top-K slices TS
SWD-1: SliceLine with observed errors p(e|C:);
[TS]] < SliceLine((Cz, Cz,.... he, hyperparameters);
2
3 SWD-2: SliceLinc with corrected errors (approximations to p(e(S.)):
4
[TS2, Quality Indicators] +
SWD-3: Combined Slices;
6
[TS] < Merge(TS; UTS));
7
return TS;
4 Proof of Concept with Synthetic Data
To demonstrate the efficacy of our algorithm fig. [ we first present evaluations on a synthetic dataset. This is done to evaluate the impact of noise on the labeling process and to determine the degree to which our algorithm can compensate for it. The synthetic data is a tabular dataset containing columns for nine "real" semanntcc dimensions for 200 000 samples each containing binary attribules. For each of the "real" dimensions (GT), a "predicted" metadata column is included as a proxy for the metadata that would be generated by CLIP in our algorithm (sce fig. [ In addition, one final column contains the binarized DuT errors (c). The first four dimensions are generated to be imbalanced with only 5% of the samples belonging to the attribute "1" The other five dimensions are generated such that both attributes have equal distribution. The error column is designed such that weak slices are induced for the specified ground-truth attributes.
We consider three regimes of noise, i.e., different quality of labeling of the simulated annotation process: (i) a regime of "good" quality CLIP labeling, represented with Pe above 80%, (ii) a regime of "medium" quality CLIP labeling, represented with pe between 40% and 70,, and (iii) regime of "bad" quality CLIP labeling, represented with Pe between 10% and 40%. For all three regimes, experiments to account for statistical influence. Further details about the dataset generation can be found in appendix A.2. In fig. 2 in the top row, the crror distributions show how labeling quality impacts the spread of error between altributes for each semantic dimension, i.e., the upper and lower ends of the bars are given by the error rates for els; el-s and similarly using or the corrected errors. In the good labeling quality regime, as expected, observed errors and corrected errors both display the same spread as the GT error. But when labeling quality is medium or bad (where the impact of eq. ( is stronger), the spread of the observed error is significantly lower than that of the corrected error. In contrast, the corrected error
considered 100 runs of the
7

---

Published in Transactions on Machine Learning Research (07 (07/2025)
either has close estimates to the true enror overestimates the true error (GT). From safety perspective, argue that overestinating the error within DNN is better than underestimating it. In the bottom row, we evaluate the results of SWD-1,2,3. This is shown by comparing how well $WD-11,,3 recover the top-k weak slices in comparison to top-k slices from Oracle, i.c. a situation where we have access to perfect "GT" labeling quality annotation. Precision and recall are calculated for the three data quality regimes w.r.t. the Oracle case by considering the overlap of identified weak slices at an increasing number of top-slices k. Note that precision and recall in this figure refer to quality metries precision and recall of the CLIP labeling. While, at level 2, the maximum number of slices & is 162 for 9 binarized dimensions we consider only slices fulfilling the cut-off requirement as weak slice. Of these 162 slices, only ~ 30 are identified as weak slices. Although under good labeling quality, the slices identified by SWD-1,2,3 basically have 100% overlap with the slices from the Oracle, under medium and strong label noise, SWD-3 shows significantly more recall than SWD-1 and marginally over SWD-2. However, this comes with a small loss precision. In cases of strong noise, SWD-1 only recovers few slices where the error signal is dominant, which explains the high precision the of low recall. SWD-3, on the other hand, has reduced precision, but recovers most of the weak slices identified by Oracle. For the rest of this work, focus primarily the slices identified by SWD-3.
weak slice discovery and not
Good Quality
Medium Quality
Bad Quality
0.30
Gl Emar pir Si Spread
Errors
ple
Conected Eror (e(S) Sproad
0.00
din
Dimensions
Precision Rocall
SND-1 Precitio
0.2
Precbam
SWD-3
0.0g
10
60
6.11
10 20
Number slices under consideration (k)
Figure 2: Based on labeling quality, we divide the generated datasets into (i) good quality (left), (ii) medium quality (middle), and (iii) bad quality (right). In three cases, look at the spread of error in GT (p(c(S)), Observed (p(elC)), and Corrected (p(e)S)). In the second row, corresponding performance in terms of precision and recall of SWD-1,2,3 are shown. Precision and Recall in this figure are metrics to evaluate weak slice recovery and are not related to labeling quality. The legend for both rows are presented on the figures on left.
5 Evaluations of real-world DNNs
In this section, we first present experimental setup. We then show the evaluation of systematic weaknesses detection method on publicly available pre-trained model for the CelebA dataset. Here, the dataset's rich metadata annotation allows to investigate the influence of noisy metadata annotation. In addition, we compare against SOTA SDM methods to evaluate our claim that adherence to the ODD descriptions useful to end users (e.g., safety experts, MT developers). Subsequently, we present the insights gained by using our approach on DNNs trained on autonomous driving datasets.
5g

---

Published in Transactions on Machine Learning Research (07/ 2225)
5.1
Experimental Setup
Datasets and Models: Five pre-trained models, ViT-B-16 Dosovitskiy et al. 2021p. 9. Faster R- R-CNN (Ren
ct al. 2015). SETR PUP
(Zheng ct al.] 2021), PanopticFCN (Li et al. 2021 and YOLOvllm (Jocher &
Qiu 2024) are evaluated using live public datasets (CelebA (Liu et al. 2015). , BDD100k (Yu et al. 2020);
Cityscapes Coodts et al.) 2016,, RailSem19 (Zendel et al. 2019 ) and EuroCity Persons dataset (Braun
et al. 2019), respectively. We restrict the number of combinations (level) to 2 in this work. However, presented in appendix B.4, our approach allows correction of errors even at higher levels of combinations. We used the cutoff for the slice error as 1.5 for all experiments except the PanopticFCN model evaluation. In the PanopticFCN evaluation, we utilize the cutoff point for the slice error as L.0ē|p as the global average error is already quite high. For a detailed experimental sctup, see appendix A.1. To foster reproducibility, code, ODDs, and the prompts used for metadata generation with CLIP are provided.
5.2 Evaluation of our Systematic Weaknesses Detection Method
Evaluating a VIT Model on CelebA dataset: As our first experiment, we evaluated the weaknesses of the ViT-B-16 (Dosovitskiy et al. 2021) model (DuT) trained on ImageNet21k (Ridnik al.] 2021). We use the model for the targeted task of identifying the class "person" in the CelebA dataset (Liu et al. 2015 as real-world proof of concept for our approach. Due to the extensive range of label categories in ImageNet (Deng ct al. 2009) and the significant noise in the labeling style, models trained on the full ImageNet datasct or its standard subset. ImageNNtllk (Russakovsky et al. 2015) can suffer from systematic weaknesses. For example, although the primary foreground object in an image might be a human, in some instances the image can be labeled as belonging to the class "person", while other similar instances the label might be about more granular classes like "bride" or "guitarist". To fix this issue, Ridnik et al. (2021) proposed 11 hicrarchies based WordNet (Miller) 1995) semantic trees such that classes at higher hierarchy levels are superclasses that subsume classes at lower hierarchy levels. However, despite these efforts, considerable label noise in terms of class overlap still persists. For example, humans holding specific objects might occur at the same hierarchy level as the class "artifact." or "person". Similar problems exist, for example, for hairstyles (see "pompadour" " existing at the same level as "person"). For a further analysis, also see the work of (Northcutt et al. 2021
Earlier works (Beyer et al. 2020 Shankar et al., 2020) have proposed using multi-label evaluation metrics a way to deal with label noise. However, we consider the simplified task of identifying dedicated class, "Person", in dataset with only human faces (CelebA) by focusing on the top-1 class predictions for level-0 of the label hierarchy proposed in ImageNet2lk. We obtain an accuracy of 94.48% on the 202 599 images in the CelebA dataset. The softmax of the top-1 prediction, see fig. 3 shows, besides the "person" class, the presence of several other classes, most prominently "artifact" and "pompadour". As this model is commonly used as a pre-trained backbone for various applications uncovering potential shortcomings might also be beneficial for potential downstream use cases. Furthermore, the CelebA dataset serves as an idcal testing ground for approaches identifying systematic weaknessos due to the availability of the ground- truth metadata attributes. As an ODD for this test case, propose simplified subset of these available metadata attributes in analogy to the work of Gannamaneni et al.] (2023). for details see appendix A.3. As proposed in our algorithm, we generate metadata using CLIP for the given ODD dimensions. In principle, stronger and more recent models such as GPT-4o (Hurst et al.) 2024) could also be used for this task at greater computational (or financial) cost. However, such models might still produce error-prone metadata. A performance comparison is given in appendix E. Subsequently, the generated motadata is combined with the errors of the DuT.
Weak Slice Discovery Since the CelebA dataset contains annotated metadata for 40 attributes, we have access to noiscless metadata which, when used with SliceLine, can be considered as the "Oracle" approach. In table [ we present the quantitative comparison of the top-7 slices identified by SWD-3 against corre- sponding slices in SWD-1 and Oracle. Basically, we list the top-7 slices of SWD-3 and evaluate where these slices would be ranked by SWD-1 and Oracle and what the corresponding statistics would be to highlight the importance of error correction. From the slice descriptions, all the identified weak slices contain some variation of the semantic concept. "wearing hat". The discovery of these slices can be seen within the context
9

---

Published in Transactions on Machine Learning Research (07/2025)
Siice Description
Slice S
Slice Statistics
Histogram of pretrained VIT-B-16 predictions on CelebA dalase!
Slice Size |S|: 7600
Peison
wearing-hat: true beard: Talse
(+[)) of Slice: 0.6856
Performance degradation: -0.6304
10'
Court
101
10'
ed fut
Ins
on
sliia
plan
hair
papar
mooo
round ail.lii
exomate
Top-1 credicted cless
Figure 3: Left: The hierarchy level-0 Riidiik et al. 2021 predictions of the pre-trained ViT-B-16 model on the full CelebA datasct converted into a binary classification problem. While majority of the predictions are correct, there is a non-trivial subset of images with systematic errors due to label overlap issues. Right: Top-1 weak slice, identified by SWD-3, of a ViT-B-16 classification model trained on ImageNet21k and evaluated on the full CelebA dataset. The statistics provide a quantitative evaluation of the entire slice. For qualitative evaluation, we provide some sample images from the slice.
of the frequent misclassification of images as class "artifact" by the DuT (see fig. In these cases, the model likely focuses on the hats the foreground object and predicts the class "artifact". To evaluate the quality of the identified slices, utilize the errors of the slices, i.e., Peorr(e(S), p(e|C), and p(elS) defined in section B We observe, based on the rank column, that the top-6 Oracle slices are captured in top-7 SWD-3 slices. Notably, while the observed error p(e(C) of SWD-1 underestimates the true error p(e(S) of Oracle, SWD-3 effectively corrects this in (e)rr(e(S). For instance, in the third row, which corresponds to the top-ranked weak slice identified by the Oracle, the difference between the Oracle slice error and the SWD-1 slice error is 0.3, while between SWD-3 and Oracle it is only 0.07. A thorough evaluation of our approach the top-60 slices shown in fig. in appendix C.1 reveals that SWD-3 obtains 100% recall of weak slices at the cost of reduction in precision. Note that precision and recall in this figure refer to quality metrics on weak slice discovery and not precision and recall of the CLIP labeling. From safety perspective, given the noisy labeling, high recall (detection of all weak slices) at the cost of some reduction in precision can be considered acceptable. Interestingly, the top-1 slice of SWD-1 (not shown in the table) refers to slice description "wearing hat: true" and "pale-skin: true". Gannamaneni et al.] (2023) discussed the limitations of CLIP in separating the latter dimension and corresponding low performance. This high level of noise in the top-1 slice while SWD-3 effectively corrects
generated metadata leads to SWD-1 identifying "palo-skin" for this by discarding the wrongly detected slice "invalid" using the quality indicators (see algorithm I and hence does not identify this dimension in top-7. For a qualitative evaluation of SWD-3, the top-1 slice with sample images from the slice are available in fig. 3 (see fig. 6 in appendix C.2 for a qualitative evaluation of the top-5 slices).
Comparison to SOTA SDM Methods: In addition to the evaluation of SWD-3, we compare three SOTA methods DOMINO (Eyuboglu et al., 2022), Spotlight (d (d'Eon et al. 2022), , and SVM FD (Jain al. 2023) against Oracle. Similarly to our work, DOMINO and SVM FD use CLIP (ViT L/14) in their workflows. However, they encode the images in the CLIP embedding space and then search for weak slices without explicitly enforcing any semantic concepts. To describe the slices, both approaches perform an additional step, where the identified slices are explained using lext. from large language models. In contrast, Spollight directly uses the embedding space the DuT to cluster weak slices and provides no descriptions of the identified slices. The former methods follow a broader trend (like us) of using foundation models like CLIP in testing smaller models. However, they do not thoroughly address limitations in CLIP's capabilities and the limitations of their approaches w.r.t. actionability when the slice descriptions are not very meaningful.
10

---

Published in Transactions on Machine Learning Research (07/2025)
Slice
Slice Description
SWD-3
SWD-1
Oracle
rank(S)
|S corr Poorr (e
(Sn
[
p(e[C)
rank(S) |S| P(e]S)
Wearing-Hat Hat: True Beard: False
Si
1
7600
0.69
6
12152
0.33
2
6267
0.51
Wearing-Haa True
S2
5132
0.60
3
8573
0.36
6476
0.45
Smiling: False
Wearing-Hate True Gender: Female
S3
3
4435
0.61
2
7393
0.38
1
2947
0.69
Wcaring-Hat: True Age: Young
St
4
7974
0.54
4
12758
0.34
6937
0.50
Wearing-Hats True
Ss
5
8606
0.54
5
12594
0.33
6
8417
0.45
Eyeglasses: False
Wearing-Hats True Goatee: False
So
6
8845
0.53
7
11453
0.33
4
8284
0.46
Wearing-Hate True
9676
0.51
8
15501
0.32
9795
0.44
Bald: False
Table 1: Evaluation of top-7 slices of SWD-3 (see algorithm 1 by comparing its statistics with corresponding slice statistics of SWD-1 and Oracle. The rank column indicates the slice ranking in each approach. The size of D for this experiment is 202 599 images.
Our approach tackles both these limitations as we address noise in CLIP labeling and also correctness of descriptions. However, our approach also has a limitation as can only identify weaknesses w.r.t. dimen- sions in while the other methods could identify more novel weaknesses. However, this advantage of SOTA methods, as will be shown below, can only be realized the description or coherence of slice is understand- able and actionable to the end-user. To assess the actionability of the SOTA methods, we consider (i) slice descriptions based on the methods themsclves, (ii) slice coherence based human inspection, and (ifi) slice coherence based on overlap with lop-5 slices of Oracle.
Qualitative results and slice descriptions are provided for the three methods in appendix C.2. We identified that DOMINO descriptions can be very generic and not helpful in identifying the unique attributes slice. This problem was also discussed in other works (Jain el al.) 2023 Gao et al. 2023). For Spotlight, descriptions are not available as part of the method. In contrast, in SVM FD, the slice description targeted and covers one dimension of the weak slice identified by Oracle, namely, "wearing hat". However, shown carlier, the weaknesses identified from Oracle stem from the combination of semantics. Therefore, slice descriptions from the SOTA methods are not enough for actionability. Second, to further evaluate the coherence of the slices, we manually inspect sample of the images from a slice to identify the semantics. Such an approach is necessary for all methods that do not provide slice descriptions. For such manual inspection to identify the coherence of the slice, we consider samples from the slice and samples from the remaining data (last column) as form of control group. For top-1 slices of all three approaches, is hard to determine what uniquely constitutes the top-1 slice when considering the combination of semantics. Furthermore, such an exercise is time intensive and might potentially uncover spurious patterns.
Finally, to evaluate the coherence of the slice based on overlap with Oracle slices, we present in table 2 the top-1 slice identified by each method, their corresponding statistics and the overlap (Jaccard Similarity Coefficient) the top-1 slice with the top-5 Oracle slices. From the slice statistics, it can be observed that the methods recover slices with significant performance degradation and observed error p(elC). However, the overlap of the top-1 slices with top-5 of Oracle is quite low. This indicates that the methods might be uncovering weaknesses w.r.t. dimensions not present in the ODD. However, without useful descriptions,
11

---

Published in Transactions on Machine Learning Research (07/2025)
Method
Slice Statistics
Slice Coherence with Attributes Overlap with Oracle top-5 slices
Overlap with FNs
Perf. degr.
Size
ølD
|S|]
p(e(C)
SP)
88) J(S S() S(S
S
[Spl
p(e|C)
DOMINO
-0.5629
11726
0.6181
0.13
0.19
0.20
0.21
0.24
0.64
Spotlight
-0.8622
4050
0.9179
0.32
0.32
0.32
0.31
0.31
0.33
SVM FD
-0.3844
2642
0.4295
0.11
0.15
0.16
0.16
0.16
0.24
Comparison of three metadata-free SOTA methods with lop-5 slices of Oracle. )( indicates the Jaccard similarity coefficient between the two slices. appendix C.2 contains samples from each slice of the SOTA methods along with slice descriptions and statistics. For overlap with the oracle slice, higher values are better. For the overlap with the FNs, low values indicate that a slice does not contain "significant" weaknesses or is highly specific, while high values indicate that potentially all weaknesses of the DuT are in one slice and it might, therefore, be too generic. This implies that in general one would expect or desire medium overlap ranges.
Table 2:
the actionability of these slices is low. Furthermore, we evaluate the overlap of the top-1 slice with a slice that is purely made up of FNs of the DuT. High values in this column might indicate that priority is given to identifying PNs rather than semantic coherence, as it is unlikely that all weaknesses of a DNN can be explained by one semantic concept. Therefore, grouping all FNs into one slice would be counterproductive. As DOMINO captures 64% of all false negatives in its top-1 slice, unlikely that such a slice is actionable. In contrast, Spotlight and SVM FD capture fewer FNs in the top-1 slice. Therefore, they might be capturing some form of combination of semantics. Based on these evaluations, we conclude that the SOTA methods, when integrated with improved slice description techniques, could complement our approach. However, in their current form, our approach offers greater actionability due to its inherent slice descriptions.
5.3 Insights on SOTA Pedestrian Detection Models
Having shown the benefits of our proposed method, we evaluate a more safety-relevant task of pedestrian detection using models trained in real-world autonomous driving (AD) datasets to identify their system- alie weaknesses when predicting the class "pedestrian". For this, we require pedestrian level performances (intersection-over-uuii) (IoU)) and metadata. To avoid noisy labeling in our metadata generation step, we perform some additional steps which were not required for the previous experiment. First, cropped all pedestrians from the images and considered these crops as D. This is done to focus the CLIP model only on pedestrians during metadata generation
Sccond, we calculate the pixel area of the pedestrians based on the ground truth bounding box area and use this to filter D by removing pedestrians that occupy small pixel areas ("smaller" pedestrians). Such filtering is necessary as: (i) Due to low resolution and high pixelation of "smaller" sized pedestrians, i.c., there is high aleatoric uncertainty regarding the correct labels affecting both CLIP and human labelers in understanding the image content (e.g., to determine gender, age, etc.). (ii) "Smaller" pedestrians are more likely to be farther from the ego-vehicle and, therefore, might be considered less safety-relevant (in terms of vehicle braking time). (iii) As the small size can be strongly correlated to performance (due to distance [Gannamancni et al. 2021 Lyssenko el. al., 2021] or occlusion), this signal can strongly dominate the search for systematic weaknesses by SliceLine, thus not providing any novel insights in terms of systematic weaknesses. For this reason, we remove the low-rcsolution "smaller" pedestrians to improve the quality of metadata generation and gain further novel insights about model failures w.r.t. more safety-relevan pedestrians.
6To avoid that the aspect ratio of pedestrians is changed the CLIP pre-processing, we use padding to obtain square crops. Unless small size is due acclusion. For BDD100k dataset, where occlusion is available as annotation, show impact. of occlusion as well
12

---

Published in Transactions on Machine Learning Research (07 2225))
The metadata generation using CLIP is performed using ODDs more suitable for automotive context (see ap- pendix A.3). We also perform manual evaluation of subset of images (n = 60) for each attribute in each dataset to evaluate the quality of the generated metadata by estimating the precision and recall re (as discussed in section 3
and show the results in table 3
Sem.
Estimated Precision PC RailSem19
Estimated Recall FC
Attri.
dim.
BDD100k
Cityscapes
BDD100k
Cityscapes
RailSem 19
Adult
0.95 = 0.03
0.99 + 0.02
0.97 = 0.02
0.76 = 0.03
0.70 + 0.02
0.55 + 0.02
Age
Young
0.69 + 0.06
0.56 + 0.06
0.42 + 0.06
0.93 + 0.06
0.97 + 0.06
0.94 + 0.06
Female
0.84 = 0.05
0.97 + 0.02
0.85 = 0.04
0.90 = 0.05
0.95 0.02
0.87 0.04
Gender
Male
0.94 = 0.03
0.97 + 0.02
0.94 = 0.03
0.88 = 0.03
0.97 0.02
0.92 = 0.03
Bright-
Cloth.-
0.81 + 0.05
0.85 + 0.04
0.79 + 0.05
0.30 = 0.05
0.23 + 0.04
0.66 + 0.05
color
color
Dark-
0.76 + 0.05
0.65 + 0.06
0.82 + 0.05
0.96 + 0.05
0.97 + 0.06
0.89 + 0.05
color
Skin-
Dark
0.82 + 0.05
0.55 + 0.06
0.56 + 0.06
0.92 =
0.71 0
0.76 + 0.06
color
White
0.99 + 0.02
0.95 + 0.03
0.89 0.04
0.96 =
0.75 + 0.04
True
0.71 0.06
0.63 + 0.06
0.87 = 0.04
0.06
0.87
0.64
Blurry
False
0.48 + 0.06
0.95 + 0.03
0.84 0.05
0.74 + 0.06
0.82 +
0.95 + 0.05
Constru.-
False
0.97 0.02
0.98 = 0.02
Worker
True
0.65 + 0.06
0.55 + 0.06
Table 3: The estimated precision and recall using our proposed approach for evaluating the quality of the generated metadata. Here, provide the mean and a o/2, for n of 60, of the estimated precision and recall. Certain dimensions like occlusion are available as part of the datasets themselves. We do not perform human- evaluation for these dimensions but these are considered in the weak slice search.
In these experiments, using SWD-3, we evaluate the weaknesses of an object detection model (Faster R-CNN) a segmentation model (SeTR PUP), and a panoptic segmentation model (Panoptic-FCN), The models are evaluated on their respective datasets, Le., BDD100K, Cityscapes, and RailSem19. Samples of image crops of the identified top-1 weak slice for each experiment are shown in fig. A (see figs. 10 to 12 in appendix C.2 for top-5 weak slices). In table 4 we present the largest. and worst performing slice of the lop-5 to provide insights about the three models. In all three experiments, the performance degradation of the identified slices is significant. "Occlusion", skin-color and clothing-color are reoccurring slice descriptions for the first two models, which are tested on datasets that contain images with many nighttime scenes (BDD100k) or relatively high gray-toned scenes (Cityscapes). In contrast, the third model, which contains relatively brighter scenes, has a significant weakness for the dimension "age". The estimated precision Pc and recall re in table g were provided as input to algorithm [ to obtain these slices and determine the quality of the identified weaknesses. Therefore, in contrast. to SOTA SDMs, our approach identiifee human-underrlanndally salety-relevant systematic weaknesses in DNNs used for real-world applications. To evaluate scalability, we also investigate systematic weaknesses in the YOLOvlIm (Jocher & Qiu 2024) model by treating it as the DuT and evaluating it on the comparatively larger EuroCity Persons (Braun et al. 2019 dataset. The results are presented in appendix D.
6 Conclusion
In this work, we present an algorithm for our Systematic Weakness Detector (SWD) to analyze the sys- tematic weaknesses of DNNs that perform classification, object detection, and semantic sogmentation tasks on image data. In the first step, we overcome the problem of missing metadata by generating meladata with a foundation model. Subsequently, in the second step, we perform slice discovery using the structured motadata, which includes the per-object performance of the DuT and the previously collected per-object metadata. Using our algorithm, we transform the slice discovery of unstructured image data into an (ap- proximate) slice discovery problem on structured data. In addition, we study the impact of noisy labeling in
13

---

Published in Transactions on Machine Learning Research (07 (07/202) 2025)
(a) Faster R-CNN
(b) SeTR
(c) Panoptic-FCC FCN
Figure 4: Left: Samples from top-1 weak slice of a Faster R-CNN object detector trained and evaluated on BDD100k datasct. Middle: Samples from top-1 weak slice of ScTR model trained and evaluated on Cityscapes dataset. Right: Samples from top-1 weak slice of Panoptic-FCN model trained and evaluated RailSem19 dataset.
a Bayesian framework and operationalize it by integrating crror correction and slice validity basod on quality indicators into our approach.
Main Results: In the ablation experiments, show that our SWD detects the same weak slices would be identified in hypothetical cases where we have access to perfect metadata. The primary advantage of our alm, in comparison to SOTA methods, is that the identified weak slices are aligned with human- understandable semantic concepts that can be derived from a description of the ODD. As upcoming safety and trustworthy Al specifications increasingly mandate evidence to support safety argumentation with re- spect specific ODDs, the results of our approach can serve as a direct contribution to these requirements. In addition, the identification of human-understandable weak slices enables ML developers to take mitiga- tion actions, such targeted acquisition generation of data, addressing the weaker slices and, thus, facilitating elfective re-training with limited acquisition budget. Furthermore, has clear advantages over several metadata-fi freee SOTA methods by giving more actionable results, and we demonstrate the applicability of our approach by identifying systematic weaknesses in multiple AD datasets. For this, we also provide a quantitative evaluation of the quality of the generated metadata.
show that our approach
On metadata generation and incompleteness of ODDs: minimum metadata labeling quality is required for the discovered slices to be meaningful. In addition to our proposed metadata quality estima- tion, future works could therefore focus on improving metadata quality by human correction of a subset of generated metadata, fine-tuning of CLIP (Fyuboglu et al.) 2022). and metadata acquisition from other sources (e.g,, depth sensor for distance). Secondly, all approaches based on ODD definitions, like ours or PromptAttack (Metzen et al.) 2023 would suffer from the lack of completeness of the semantic concepts
Largest Slice (in top-5)
Worst Performing Slice
Model & Dataset
(in top-5)
LS| 2
%
Poorr (e(S)
Perf.
[S] D
%
Peorr(e/3)
Perf. Degr.
Degr.
Faster R-CNN
34.44%
0.1263
-0.0693
14.22%
0.2206
-0.1636
BDD100k
SeTR
13.34%
0.0594
-0.0446
9.24%
0.1046
-0.0897
Cityscapes
Panoptic-FCN RailSem19
25.49%
0.8663
-0.222
8.13%
1.0
-0.4602
Table 4: Quantitative analysis of three pre-trained autonomous driving models (results are only for SWD-3). From the top-5 weak slices, we show the largest. slice and the weakest performing slice. Please refer to the appendix C.2 for the top-5 slices.
14

---

Published in Transactions on Machine Learning Research (07/2025)
in the ODD. A potential solution could be in the direction of Gannamaneni et al. (2024) by performing root-cause analysis of found weaknesses. Such approaches could address potential issues between correlation and causation for found small slices.
On generation of ODDs: ODDs are a critical component in the functioning of our algorithm, necessi- tating careful consideration of their availability. In safety-critical domains such as automotive, medical, and aerospace applications, we posit that domain and safety experts possess the requisite expertise to develop ODDs. In contrast, for use cases lacking predefined ODDs, the use of large language models (LLMs) to sup- port ODD generation presents a promising direction, akin to the tag generation approach proposed in (Chen ct al. 2025). Nevertheless, we emphasize that human oversight remains essential, both in the final approval of the generated ODDs and in their subsequent refinement and extension.
On considering foundation models as DuT: In this work, we evaluate of (SOTA) architec- tures for classification, segmentation, and object detection tasks within the framework of DuT, leveraging more "powerful" foundation model (CLIP) as meladata generation function. However, the proposed algorithm can also be employed to identify systematic weaknesses and biases inherent in foundation models themselves, by treating the foundation model as the DuT. For instance, if the foundation model is intended for tumor detection in X-ray scans, then suitable choices of 2 and DD must be determined to enable the application of the algorithm.
This work focuses on false negatives for two main reasons: first, in safety-critical tasks, mitigating false negatives generally more crucial than addressing false positives; second, the ODDs we develop, based on input from domain experts, are specifically centered around pedestrian-related semantics and their attributes. Addressing false positives would require expanding the ODDs to encompass all non- pedestrian classes that could potentially be misclassified. This challenge is both an open-set problem and highly dependent on the specific DuT under consideration. If false positives are to be incorporated within our algorithm, an initial exploratory analysis would be helpful to understand their distribution. For example, in the RailSem19 experiment, we observed that a large portion of false positives involved railway signs being misclassified as redestrians. Based on such analyses, new ODDs (e.g., based on object type, visual features like color, etc.) could be developed for these frequently misclassified objects, after which our algorithm could be applicd. The exploratory analysis could be based on concept-focused clustering approaches, compare, e.g.: Haedecke et al. (2024), or even use the outcome of slice-and-Lag approaches. Nevertheless, even when focusing only on false negatives, our approach provides meaningful insights into, often more critical, missed detections in terms of human-understandear
On false positives:
and, thereby, actionable weak slices. We believe that such results can contribute to the development of trustworthy AI models.
7 Acknowledgments
This work has been funded by the European Union and the German Federal Ministry for Economic Affairs and Climate Action as part of the tl project.
References
Martin Atzmueller. Subgroup discovery. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 5(1):35-48) 49, 2015.
Lucas Beyer, Olivier Hénaff, Alexander Kolesnikov, Xiaohua Zhai, and Aäron van den Oord. Are we done with Imagenet? ar preprint 2xx::2006.07159, 2020.
Markus Braun, Scbastian Krebs, Fabian B. Flohr, and Dariu M. Gavrila. Eurocity persons: A novel bench- mark for person detection in traffic scenes. IEEE Transactions on Patlern Analysis and Machine Intelli- gence (TPAMT), pp. 1-1, 2019. ISSN 0162-8828. doi: 22110///PAM..221122577777
Joy Buolamwini and Timnit Gebru. Gender shades: Intersectional accuracy disparities in commercial gender classification. Tn Conference on Fairness, Accounlability and Transparency, pp- 77-91. PMI.R, 2018.
15

---

2022. doi: 10.1007/9988-3-03--00 Published in Transactions on Machine Learning Research (07/2025)
Simon Burton, Christian Hellert, Fabian Hüger, Michael Mock, and Andreas Rohatschek. Safety Assurance of Machine Learning for Perceplion Funclions, pp.
Springer International Publishing, Cham,
12. URL
Muxi Chen, YU LI, and Qiang Xu. HiBug: On human-interpretable model dcbug. In Ad- vances in Newral Information Processing Systems (NewrIPS), volume 36, pp. 4753 4766. Curran Associates, Inc., 2023. URL https::/proceeddngs.
/
Muxi Chen, Chenchen Zhao, and Qiang Xu. HiBug2: Efficient and interpretable error slice discovery for comprehensive model debugging. In The Thirleenth International Conference on Learning Representalions (ICLR), 2025. URL https:/
Yeounoh Chung, Tim Kraska, Neoklis Polyzotis, Ki Hyun Tae, and Steven Euijong Whang. Slice Finder: Automated data slicing for model validation. In 2019 IEEE 35th International Conference on Data Engi- neering (ICDE), pp. 1550-1553, IEEE, 2019.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understand- ing. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 3213 -3223, 2016.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Proceedings of the IEEE Conference on Computer Vision and pattern Recognation (CVPR), pp. 248 255, 2009.
Greg d' d'Eon, Jason d'Eon, James R Wright, and Kevin Leyton-Brown The spotlight: A general method for discovering systematic errors in deep learning models. In 2022 ACM Conference on Fairness, Account- ability, and Transporency, pp. 1962-1981, 2022.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In Inter- national Conference on Learning Representations (ICLR), 2021. URL https://openrevicen net/forum? id=YicbFaNTTy
European Union Aviation Safety Agency. EASA artificial intelligence concept paper issue 2: Guidance for level 1 & 2 machine learning applications. Concept Paper Issue 2, European Union Aviation Safety Agency (EASA), Cologne, Germany, 03 2024. URL https: easa httpa//wwrraaaaaaaaaa//ooeeeaaaa -atpr///ennonaaaaaaaaaaaacaaaaaa
Sabri Eyuboglu, Maya Varma, Khaled Kamal Saab, Jean-Benoit Delbrouck, Christopher Lee-Messer, Jared Dumnmon, James Zou, and Christopher Re. Domino: Discovering systematic errors with cross-modal embeddings. In International Conference on Learning Representations (ICLR), 2022. URL https:// wtwp::/.w.....ooooaddddddooooooo
Sujan Gannamaneni, Sebastian Houben, and Maram Akila. Semantic concept lesting in autonomous driving by extraction of object-level annotations from carla. In Proceedings of the TEEE/CV. International Conference Computer Vision (ICCV) Workshops, pp. 1006 1014, October 2021.
Sujan Sai Gannamaneni, Arwin Sadaghiani, Rohil Prakash Rao, Michael Mock, and Maram Akila. In- vestigating clip performance for meta-data generation in ad datasets. In Proceedings of the IEFE/CVP Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pp. 3840 18440-850,, June 2023.
Sujan Sai Gannamaneni, Michael Mock, and Maram Akila. Assessing systematic weaknesses of DNNs using counterfactuals. Al and Ethics, pp. 27-35, 2024. doi: 10.0077/44881-033--000000 URL https
16

---

Published in Transactions on Machine Learning Research (07/2025)
Irena Gao, Gabriel Ilharco, Scott Lundberg, and Marco Tulio Ribeiro. Adaptive testing of computer vision models. In Proceedings of the IEEE/CVF International Conference on Compuler Vision (CVPR), pp. 4003 4014, 2023.
Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. Shorteut learning in deep neural networks. Nature Machine Intelli- gence, 2(11):665 673, 2020.
Riccardo Guidotti, Anna Monreale, Salvatore Ruggieri, Francesca Naretto, Franco Turini, Dino Pedreschi, and Fosca Giannotti. Stable and actionable explanations of black-box models through factual and coun- terfactual rules. Data Mining and Knowledge Discovery, pp. 1-38, 2022. doi: 10.0007/s100188422200885--
Elena Hacdecke, Maram Akila, and Laura von Rueden. Towards linking local and global explanations for AI assessments with concept explanation elusters. Proceedings of the AAAT Symposium Series, 4(1):106-109 Nov. 2024. doi: 10.1609/aaaie v411.31779. URL htttts://tssss../ aaai. wergg/nddx.pppe httppp/AA-S///raii.....c view/31779
Martin Herrmann, Christian Witt, Laureen Lake, Stefani Guneshka, Christian Heinzemann, Frank Bonarens, Patrick Feifel, and Simon Funke, Using ontologies for dataset engineering in automotive AI applications. In 2022 Design, Automation and Test in Europe Conference and Exhibition (DATE), pp. 526-53) 2022. doi: 10.23919 0TB.I///DDEEE1114220009779/
Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. GPT-4o system card, arXiv preprint 14.2:24410221266 2024.
ISO. TSO/PAS 8800:2024 Road vehicles Safety and artificial intelligence, 2024. Available at: https: 7/www. iso. org/ standrdd/3dddd........ html
Saachi Jain, Hannah Lawrence, Ankur Moitra, and Aleksander Madry. Distilling model failures as directions in latent space. In The Eleventh International Conference Learning Representations (ICLR), 2023. URL https::/opeererv// wtt.://ooommmm///-99////yy
Glenn Jocher and Jing Qiu. Ultralytics yololl, 2024. URL httpgs//ggttuub..commalllaaaall
ultralytics
Philip Koopman and Frank Fratrik. How many operational design domains, objects, and events? In SafeAI@AAAI, 2019.
Salyapriya Krishna, Tessa Han, Alex Gu, Steven Wu, Shahin Jabbari, and Himabindu Lakkaraju. The disagreement problem in explainable machine learning: A practitioner's perspective. Transactions on Machine Learning Research (TMLR), 2024. ISSN 2835-8856. URL http:::://ppprreeee.eeee/fo/oooe
JESY2WTZCe
Xinyue Li, Zhenpeng Chen, Jie M Zhang, Federica Sarro, Ying Zhang, and Xuanzhe Liu. Dark-skin indi- viduals are at more risk on the street: Unmasking fairness issues of autonomous driving systems. anXiv preprint ar Xi1:22300. 02935, 2023.
Yanwei Li, Hongshuang Zhao, Xiaojuan Qi, Liwei Wang, Zeming Li, Jian Sun, and Jiaya Jia. Fully convo- lutional networks for panoptic segmentation. In Proceedings of the IEEE/CVP Conference on Compuler Vision and Pattern Recognition (CVPR), pp. 214-222 223, 2021.
Zhuang Liu, Hanzi Mao, Chao- Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In Proceedings of the IEEE/CVF' Conference on Computer Vision and Pattern Recognition (CVPR), pp. 11976-11986, 2022.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
17

---

Published in Transactions on Machine Learning Research (07/2025)
Scott M Lundberg and Su-In Lee. A unified approach to interpreting model predictions. In Aduances in Neural Information Processing Systems (NeurlPs), volume 30. Curran As-
sociates, Inc., 2017. URL https: https://proceeddngg ittp://prrreeddg...rrr.r.pppppp www.ipp.cc/papprrrr. http://aaprr..ilssepppre//////// Filee///apee/200////ll/ 62008821176222166551886677787718
pdf
Maria Lyssenko, Christoph Gladisch, Christian Heinzemann, Matthia Woehrle, and Rudolph Triebel. From evaluation to verification: Towards task-oriented relevance metries for pedestrian detection in safety- critical domains. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recogni- tion (CVPR) Workshops, pp. 38-45, June 2021.
Usman Mahmood, Robik Shrestha, David DB Bates, Lorenzo Mannelli, Giuseppe Corrias, Yusuf Emre Erdi, and Christopher Kanan. Detecting spurious correlations with sanity tests for artificial intelligence guided radiology systems. Frontiers in Digital Health, 3:671015, 2021.
Jan Hendrik Metzen, Robin Hutmacher, N. Grace Hua, Valentyn Boreiko, and Dan Zhang. Identification of systematic errors of image classifiers on rare subgroups. In Proceedings of the IEEE/CVE International Conference on Computer Vision (ICCV), pp. 5064-5073, October 2023.
George A Miller. Wordnet: A lexical database for english. Communications of the ACM, 38(11):39 41, 1995.
Curtis Northcutt, Lu Jiang, and Isaac Chuang. Confident learning: Estimating uncertainty in dataset labels. Journal of Arlificial Intelligence Research (JAIR), 70:1373 1411, 2021.
Luke Oakden-Rayne Jared Dunnmon, Gustavo Carneiro, and Christopher Ré. Hidden stratification causes clinically meaningful failures in machine learning for medical imaging. In Proceedings of the A ACM confer- ence on Health, Inference, and Learning, pp. 151 159, 2020.
Gregory Plumb, Nari Johnson, Angel Cabrera, and Amoct Talwalkar. Towards a more rigorous science of blindspot discovery in image classification models. Transactions on Machine Learning Research (TMLR), 2023. ISSN 2835-8856. URL
Expert Certification.
Alec Radford, Jong Wook Kim, Chris Hallacy Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Proceedings of the 38th International Conference on Machine Learning (ICML), volume 139 of Proceedings of Machine Leurning Research, pp. 8748 8763. PMLR, 18 24 Jul 2021. URL htttps://proceeteee
mlr wwt..meeeeeee////eaeeaeeeeeeeee prees//113//raaford21a. html
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster R-CNN: Towards real-time ob- ject. detection with region proposal networks. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett (eds.), Advances in Newral Information Processing Systems, volume 28. Curran Associates, Inc., 2015. URL https:// www.ggs...uiis..mo cc/paper files//appor2015/EiI. pdf
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 'Why should trust you?": Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '16, pp. 1135-1144, New York, NY, USA, 2016. Association for Computing Machinery. ISBN 4703430322 doi: 10. 01.114//2039972.2333777 URL https:///tpt..oo org/
10. 1 21111/99///662222 2939778
Tal Ridnik, Emanuel Ben- Asaf Noy, and Lihi Zelnik-Manor. Imagenet-21k pretraining for the masses. In Shirty-fifth Conference on Newral Information Processing Systems Datasets and Benchmarks Track (Round 1), 2021. URI. htttp:://ppenrviiie htt Vc26ol
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. Imagenet large scale visual recognition challenge. International Journal of Computer Vision 115 115(3) December 2015. ISSN 1573-1405. doi: 10. 10.1007 12000/111263--1-081666 URL httpps://aaa.orrr//..ooo////111/
18

---

Published in Transactions on Machine Learning Research (07/)025)
Svetlana Sagadceva and Matthias Bochm. Sliceline: Fast, Imear-algebra-bbsse slice finding for ML model debugging. In Proceedings of the 2021 International Conference on Management of Dala, pp. 2021.
Shiori Sagawa*, Pang Wei Koh*, Tatsunori B. Hashimoto, and Percy Liang. Distributionally robust neu-
ral networks. In International Conference Learning Representations (ICLR), 2020. URL https:
Shibani Santurkar, Dimitris Tsipras, and Aleksander Madry. (BREEDS): Benchmarks for subpopulation
shift. In International Conference on Learning Representations (ICLR), 2021. URL https://ppeerreviie
Vaishaal Shankar, Rebecca Roelofs, Horia Mania, Alex Fang, Benjamin Recht, and Ludwig Schmidt, Eval- uating machine accuracy on ImageNet. In Proceedings of the 37th International Conference on Machine Learning (ICML), volume 119 of Proceedings of Machine Learning Research, pp. 8634 8644. PMLR, 13 18 Jul 2020. URL htttss://proceeddngg mlr wwwpssssssl///peeep http11/99/aaaa22...cc hhtmll
Eric Slyman, Minsuk Kahng, and Stefan Lee. VLSlice: Interactive Usion-and-languaee slice discovery. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 15291-15301, 2023.
Zeyu Wang, Klint Qinami, loannis Christos Karakozis, Kyle Genova, Prem Nair, Kenji Hata, and Olga Russakovsky. Towards fairness in visual recognition: Effective strategies for bias mitigation. In Proceedings of the IEEE/CVE Conference on Computer Vision and Paltern Recognition (CVPR), pp.
2020.
Olivia Wiles, Isabela Albuquerque, and Sven Gowal. Discovering bugs in vision models using off-the-sheff image generation and captioning, In NewrlPS ML Safety Workshop, 2022. URL https://openrevier.
net/forum? mamoormmm WO1D
Kai Yuanqing Xiao, Logan Engstrom, Andrew Ilyas, and Aleksander Madry, Noise signal: The role of image backgrounds in object recognition. In International Conference on Learning Representations (ICLR), 2021. URL httpss// http::///.ppppooeeeeoo///ooo/oaa xX77xg
Fisher Yu, Haofeng Chen, Xin Wang, Wenqi Xian, Yingying Chen, Fangchen Liu, Vashisht Madhavan, and Trevor Darrell. BDD100k: A diverse driving dataset for heterogeneous multitask learning. In Proceedings of the /EE/ Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2636-2645, 2020.
Mare Zeller, Thomas Waschulzik, Martin Rothfelder, and Cornel Klein. Safety assurance of a driverless regional train-insight in the safe.trAIn project. In 2023 IEEE 34th International Symposium on Software Reliability Engineering Workshops (ISSREW), pp. 41-42. IEEE, 2023.
Oliver Zendel, Markus Murschitz, Marcel Zeilinger, Daniel Steininger, Sara Abbasi, and Csaba Beleznai. Railsem19: A dataset for semantic rail scene understanding. In Proceedings of the IFFE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, June 2019.
Sixiao Zheng, Jiachen Lu, Hengshuang Zhao, Xiatian Zhu, Zekun Luo, Yabiao Wang, Yanwei Fu, Jianfeng Feng, Tao Xiang, Philip H.S. Torr, and Li Zhang. Rethinking semantic segmentation from a sequence- Lo-sequence perspective with transformers. In Proceedings of the IEEE/CVF Conference on Compuler Vision and Pattern Recognétion (CVPR), pp.
June 2021.
19


Reviews:
[{"review_id": "KwCSubMKBC", "reviewer": "Reviewer_CMrK", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. The addressed problem is important: to provide more interpretability for DNNs and ensure their safety.\n\n2. The paper writing is easy to follow: the storyline is very clear, and the presentation is good.\n\nWeaknesses:\n\n1. Evaluated models and datasets are not state-of-the-art: the models evaluated are from 2015 to 2021, and datasets are from 2015 to 2020 (page 7). Can newer models and datasets also be considered?\n\n2. Scaling possibility can be discussed: Following weakness 1, how good is the proposed method in terms of scaling to very large models and very large datasets?\n\n3. Why CLIP for annotation: The authors mention that \"CLIP achieved SOTA level zero- shot performance on different dimensions (page 5)\", the reference points to Gannamaneni et al. (2023). If I understand it correctly, the SOTA results are by 2023. Is CLIP still SOTA now? If not, do alternatives exist?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors address the problem of systematically finding weaknesses of DNNs by using foundation models with a novel algorithm to annotate metadata. The approach is particularly useful in dealing with noise in data annotation and does not require extensive human efforts. To validate their approach, they conduct experiments on both synthetic and realistic CV datasets.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Address whether new models/datasets can be used.\n\n2. Discuss about scaling possibilty of the proposed method.\n\n3. Discuss more about why CLIP is used for annotaion.", "broader_impact_concerns": "I do not have any concern.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "IpBSDFVy7K", "reviewer": "Reviewer_wGv7", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The proposed approach offers several strengths. It identifies systematic weaknesses of deep vision models along human-understandable dimensions defined by the operational design domain (ODD), making the results interpretable and actionable. By leveraging CLIP for zero-shot image classification, the method avoids the need for manually annotated metadata, allowing it to scale efficiently to large image datasets. It also accounts for noisy or imperfect metadata, increasing robustness in practical settings.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper presents a method for identifying systematic weaknesses in vision models by discovering underperforming data slices that align with human-understandable dimensions defined in the operational design domain (ODD). To achieve this, the authors combine zero-shot semantic labeling using CLIP with a modified version of the SliceLine algorithm for slice discovery. The proposed approach also includes a mechanism to handle noise in the generated metadata, improving robustness when CLIP labels are suboptimal. The method is validated on both synthetic and real-world datasets, and is used to analyze several pre-trained DNNs, revealing interpretable and relevant failure patterns.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "In Section 2, it would be helpful if the authors described the existing SDM methods using the same notations introduced in the proposed method (Section 3). This would make it easier to clearly understand the distinctions between prior approaches and the proposed DuT framework. Additionally, it would be interesting to know whether previous slice discovery methods have attempted to address the impact of noise from a Bayesian perspective. If such approaches exist, comparing them with the proposed Bayesian formulation for handling noisy metadata could strengthen the contribution and provide further insight. Furthermore, to enhance the practical impact of the work, the authors could consider incorporating methods to improve metadata quality, such as lightweight human-in-the-loop corrections or fine-tuning foundation models like CLIP on domain-specific datasets.", "broader_impact_concerns": "As a reviewer, I do not see any immediate ethical concerns that would necessitate a detailed Broader Impact Statement beyond what is typically expected. The proposed method focuses on analyzing model weaknesses along human-understandable dimensions, which could contribute positively to improving the safety and robustness of vision systems, particularly in safety-critical applications.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "iHKhz1UF8N", "reviewer": "Reviewer_56t6", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**:\n\n- This paper provides a strong motivation for identifying samples where the network’s performance falls below the average test performance. By doing so, one can potentially improve the network’s stability and robustness during training.\n\n- The approach allows for pinpointing slices with semantics aligned with human understanding concepts, revealing the areas where the network’s weaknesses lie, offering opportunities for improvement. This approach is particularly valuable for developing better networks or emphasising the slice data to improve the performance, especially in safety-critical domains like autonomous driving.\n\n- Extensive experiments demonstrate the effectiveness of this approach in identifying slices where the network shows weaknesses. \n\n**Weaknesses**:\n- A clearer explanation of the generated meta-data would be helpful. It’s not entirely clear to me how CLIP is used and how they differ from the ODD data.\n- This paper misses comparisons to other methods that directly use images as unstructured data, such as PromptAttack.  A comparison with these methods could potentially strengthen the need for the step of converting unstructured image data into structured data. \n- It would be better to make it clear earlier that $\\mathcal{Z}$ are the ODD information. \n- This approach is based on the need for ODD. However, how can it handle the case where no ODD are available? \n- The last sentence on page 4 is not clear to me: „As such, it therefore provides the coherence of the slices discussed above.“\n-  How does this approach address the possibility of systematic false positives?\n - How can the biases from pretrained models and foundation models be identified and potentially overcome? A discussion of this potential harm would be needed. \n\n*Clarity*:\n\n- In my opinion, a rewrite of the „metadata generation“ section is needed. It’s unclear and feels like an overstatement of the sophistication of simply using CLIP for zero-shot classification. The explanation of what the metadata are is still not clear. \n- The section of the Bayesian framework contains unclear and loose statements. For example, in my understanding, $p(e|S)$ represents the expected error given the samples in slice $S$, based on the ground truth target from the dataset $\\mathcal{D}$. On the other hand, $p(e|C)$ represents the expected error given the samples from slice $S$, but calculated using the prediction of CLIP instead of the DNN. If this is correct, then this section switches between the usage of „expected error“,„true error,“ and the correct usage of probabilities and actual errors. A rewrite of this section is needed to provide a proper mathematical description and usage of the individual notations.\n- Page 8, Section 5.2: The first two paragraphs could be shortened and combined, since the first paragraph feels rather distracting from what this experimental setup is actually doing here.\n\n*Questions*: \n- Section 5.1: What is the cutoff for the slice error? \n- Section 5.1 Are the ODD the target attributes from the CelebA dataset? \n-  Table 1: Was the complete CelebA dataset used as test-data for DuT? In any case, an inclusion of the portion of the slice to the actual data $\\mathcal{D}$ would be helpful. \n-  Table 2: How large is the overlap between the slices found with SWD-3 and the oracle? \n- How bad/weak are the actual predictions of the slice data?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes to use a combination of a foundation model (CLIP) for zero-shot classification to generate semantic meta-data with a slice search method (SliceLine) for searching slices of data, for which a trained network fails in predicting the correct target.  The combination of a foundation model and the Slice search method allows for finding slices which are in line with human understanding. In addition, this paper includes a step to handle the uncertainty in prediction of the foundation model in the classification tasks. Several extensive experiments on synthetic data as well as image data for classification and detection show improvements against other methods searching for slices using structured data.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- It would be helpful to provide a clearer explanation of why converting to structured data is superior to using approaches with unstructured data. \n- A rewrite of the sections mentioned (in section 3) is needed for better clarity. \n- A clearer description of the metadata is would be helpful. \n- How could the approach be adapted if no ODDs are provided? Could these be generated using an LLM? \n- A deeper  discussion on how to handle false positive systematic error is missing.\n- How can biases from pretrained models and foundation models be identified and potentially overcome?", "broader_impact_concerns": "As mentioned in the weaknesses two important discussion points would are needed:\n- A deeper discussion on how to handle false positive systematic error is missing.\n- How can biases from pretrained models and foundation models be identified and potentially overcome?", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)