Domino: Discovering Systematic Errors with Cross-Modal Embeddings
Authors: Sabri Eyuboglu
Maya Varma
Khaled Kamal Saab
Jean-Benoit Delbrouck
Christopher Lee-Messer
Jared Dunnmon
James Zou
Christopher Re
Venue: ICLR 2022 Oral (2022)
Keywords: robustness
subgroup analysis
error analysis
multimodal
slice discovery

Abstract: Machine learning models that achieve high overall accuracy often make systematic errors on important subsets (or slices) of data. Identifying underperforming slices is particularly challenging when working with high-dimensional inputs (e.g. images, audio), where important slices are often unlabeled. In order to address this issue, recent studies have proposed automated slice discovery methods (SDMs), which leverage learned model representations to mine input data for slices on which a model performs poorly. To be useful to a practitioner, these methods must identify slices that are both underperforming and coherent (i.e. united by a human-understandable concept). However, no quantitative evaluation framework currently exists for rigorously assessing SDMs with respect to these criteria. Additionally, prior qualitative evaluations have shown that SDMs often identify slices that are incoherent. In this work, we address these challenges by first designing a principled evaluation framework that enables a quantitative comparison of SDMs across 1,235 slice discovery settings in three input domains (natural images, medical images, and time-series data).
Then, motivated by the recent development of powerful cross-modal representation learning approaches, we present Domino, an SDM that leverages cross-modal embeddings and a novel error-aware mixture model to discover and describe coherent slices. We find that Domino accurately identifies 36% of the 1,235 slices in our framework -- a 12 percentage point improvement over prior methods. Further, Domino is the first SDM that can provide natural language descriptions of identified slices, correctly generating the exact name of the slice in 35% of settings. 

Published as a conference paper at ICLR 2022
DOMINO: DISCOVERING SYSTEMATIC ERRORS WITH CROSS-MODAL EMBEDDINGS
Sabri Eyuboglu'. Maya Varma", Khaled , Jeann-Bnnit Delbrouck, Christopher
Lee-Messer, Jared Dumnmon, James Zou, Christopher Ré
Stanford University, USA; {eyubog] u, mvarma2, sttabbeestaoood.ee.. (" F'Equal contribution)
ABSTRACT
Machine learning models that achieve high overall accuracy often make system- atic errors on important subsets (or slices) of data. Identifying underperforming slices is particularly challenging when working with high-dimensional inputs (e.g. images, audio), where important slices are often unlabeled. In order to address this issue, recent studies have proposed automated slice discovery methods (SDMs), which leverage learned model representations to mine input data for slices on which a model performs poorly. To be useful to a practitioner, these methods must identify slices that are both underperforming and coherent (i.e. united by cumma--understnddabe concept). However, no quantitative evaluation framework currently exists for rigorously assessing SDMs with respect to these criteria. Ad- ditionally, prior qualitative evaluations have shown that SDMs often identify slices that are incoherent. In this work, we address these challenges by first designing a principled evaluation framework that enables a quantitative comparison of SDMs across 1,235 slice discovery settings in three input domains (natural images, med- ical images, and time-series data). Then, motivated by the recent development of powerful cross-modal representation learning approaches, we present Domino, an SDM that leverages cross-modal embeddings and novel error-aware mixture model to discover and describe coherent slices. We find that Domino accurately identifies 36% of the 1,235 slices in our framework provement over prior methods. Further, Domino is the first SDM that can provide natural language descriptions of identified slices, correctly generating the exact name of the slice in 35% of settings.
12 percentage point im-
INTRODUCTION
Machine learning models often make systematic errors on important subsets (or slices) of date | For instance, models trained to detect collapsed lungs in chest X-rays have been shown to make predictions based on the presence of chest drains, a device typically used during treatment (Oakden- Rayner et al., 2019). As a result, these models frequently make prediction eтrors on cases without chest drains, critical data slice where false negative predictions could be life-threaten tffe-threatening.. Simi- lar performance gaps across slices have been observed in radiograph classification (Badgeley et al.) 2019 Zech al. 2018, DeGrave et al. 202T), melanoma detection (Winkler et al., (019), natu- ral language processing (Orr et al., 2020: Goel et al., 202I), and object detection (de Vries et al., 2019). among others. underperforming slices can be accurately identified and labeled, we can then improve model robustness by either updating the training dataset or using robust optimization techniques (Zhang et al. 2018 Sagawa et al. 2020).
However, identifying underperforming slices is difficult in practice. When working with high- dimensional inputs (e.g. images, time-series data, video), slices are often "hidden". meaning that they cannot easily be extracted from the inputs and are not annotated in metadata (Oakden-Raynee et al. 2019). For instance, in the collapsed lung example above, the absence of chest drains challenging to identify from raw image data and may not be explicitly labeled in metadata. In this setting, we must perform slice discovery: the task of mining unstructured input data for semantically meaningful subgroups which the model performs poorly.
We define data slice group of data examples that share an attribute characteristic.

---

Published as a conference paper at ICLR 2022
Evaluation Setting
Damino
Task: Datermine if the image contains a bird
Embed with cross-modal embeddingg
Cross-Modal Represennattin Space
Correlation: The presence o' a bird corelltee blue sky
bird
Irput Eneader
Data:
X
X
Thel Eneader
dog
Slice with erronaware mixture model
Q
dloud
sky
error
Explain natural language
siice
skibs without birdss
Figure 1: Proposed Approach. (Left) We design evaluation framework to systematically com- pare SDMs across diverse slice settings. Here, the example slice setting includes dataset that displays strong correlation between the presence of birds and skies. (Right) A classifier trained to detect the presence of birds makes false positive predictions on skies without birds. We present Domino, novel SDM that uses cross- moosl embeddings identify and describe the error slice.
In modern machine learning workflows, practitioners commonly attempt slice discovery with com- bination of (eature-based interpretability methods (e.g. GradCAM, LIME) and manual inspection (Selvaraju et al.) 2017 Ribeiro et al.] 2016). However, these approaches are time-consuming and susceptible to confirmation bias (Adebayo et al. 2018). As a result, recent works have proposed automated slice discovery methods (SDMs), which use learned input representations to identify se- mantically meaningful slices where the model makes prediction errors (d Eo' et al.. [2021] Yeh et al.) 2020a; (Sohoni et al. 2020 Kim et al. 2018 Singla et al. 2021). An ideal SDM should automat- ically identify data slices that fulfill two desiderata: (a) slices should contain examples on which the model underperforms, or has a high error rate and (b) slices should contain examples that are coherent, or align closely with a human-unde -nnaann-nnrtttnaablee concept. An SDM that able to reliably satisfy these desiderata across wide range of settings has yet to be demonstrated for two reasons:
No quantitative evaluation framework exists for measuring performance of SDMs with respect to these desiderata. Existing SDM evaluations are either qualitative (d'Eon et al., (02T), performed on purely synthetic data (Yeh et al. 2020aa), or consider only a small selection of tasks and slices (Sohoni et al.) 2020). A comprehensive evaluation framework should be quantitative, use realistic data, cover broad range of contexts, and evaluate both underperformance and coher- ence. Currently, no datasets or frameworks exist to support such an evaluation, making it difficult to evaluate the tradeoffs among prior SDMs.
Issue 1:
Issue 2: Prior qualitative evaluations have demonstrated that existing SDMs often identify slices that are incoherent. A practically useful SDM should discover coherent slices that are understand- able by domain expert. For example, in the chest X-ray setting described earlier, the slice "patients without chest drains" " is meaningful to a physician. Slice coherence has previously been evaluated qualitatively by requiring users to manually inspect examples and identify common attributes (d Eon eet al.) | Ych et al. (2020a). Such evaluations have shown that discovered slices often do not align with concepts understandable to a domain expert. Additionally, even if slices do align well with con- cepts, it may be difficult for humans to identify the shared attribute. Thus, an ideal SDM would not only output coherent slices, but also identify the concept connecting examples in each slice.
In this work, we address both of these issues by (1) developing framework to quantitatively evaluate the effectiveness of slice discovery methods at scale and (2) leveraging this framework to demon- cross-modal embeddings can be used to create an
strate that powerful class of accetly-develope SDM that satisfies the above desiderata. Our approach Domino identifies coherent slices and generates automated slice descriptions.
After formally describing the slice discovery problem in Section 2, we introduce an evaluation framework for rigorously assessing SDM performance in Section 3. We curate set of 1,235 slice discovery settings, each consisting of real-world dataset, a trained model, and one or more "ground truth" slices corresponding to concept in the domain. During evaluation, the SDM is provided with the dataset and the model, and we measure if the labeled slices can be successfully identified. We find that existing methods identify "ground truth" slices in no more than 23% of these settings.
2

---

Published as a conference paper at ICLR 2022
Motivated by the recent development of large cross-modee representation learning approaches (e.g. CLIP) that embed inputs and text in the same latent representation space, in Section we present Domino, novel SDM that uses cross- mossll embeddings to identify coherent slices. Cross-modie representations incorporate semantic meaning from text into input embeddings, which we demon- strate improve slice coherence and enable the generation of slice descriptions. Domino embeds inputs alongside natural language with cross-modal representations, identifies coherent slices with an error-aware Gaussian mixture model, and generates natural language descriptions for discovered slices. In Section 5, we use our evaluation framework to show that Domino identifies 36% of the "ground truth" coherent slices across three input domains (natural images, medical images, and time-series)- 12 percentage point improvement over existing methods.
RELATED WORK
Slice performance gaps. Machine learning models are often limited by the presence of underper- forming slices. In Section A.2 we provide survey of underperforming slices reported by prior studies across a range of application domains and slice types. Baadeen-Rayn et al. (2019) referred to this problem as "hidden stratification" and motivated the need for slice discovery techniques.
Slice discovery. Prior work on slice discovery has predominantly focused on input datasets with rich structure (e.g. tabular data) or metadata, where slicing can generally be performed with predicates (e.g. nationality = Peruvian) (Chung et al., 2019] [Sagadeeva & Bochm 2021 Chen et al.) 202I). The slice discovery problem becomes particularly complex when input data lacks explicit structure (e.g. images, audio, etc.) and metadata, and three recent studies present methods for performing slice discovery in this unstructured setting (d (d'Eon et al.) 2021 (Shooni et al., [2200] Kim al. 2018
Singla et al. 202T). The proposed SDMs follow two steps: (1) embed input data in a represen- tation space and (2) identify underperforming slices using clustering dimensionality reduction techniques. These SDMs are typically evaluated by measuring performance over a limited number of slice settings or by performing qualitative assessments. The trade-offs between these SDMs have not been systematically compared, and as result, the conditions under which these SDMs succeed at identifying coherent slices remain unclear.
Benchmark datasets for machine learning robustness. Recently, several benchmark datasets have been proposed for evaluating the performance of models on dataset shifts. These benchmarks are valuable because they provide labels specifying important slices of data. However, these datasets do not suffice for systematic evaluations of SDMs because they either only annotate a small number of slices (Koh et al. (222T) or do not provide pretrained models that are known to underperform on the
slices (He et al. 2021 Khosla et al.) 2011 Hendrycks & Dietterich, 2019 Liang & Zou 202I
Cross-modal embeddings. Cross-modal representation learning approaches, which embed input data and text in the same representation space, yield powerful embeddings that have contributed to large performance improvements across information retrieval and classification tasks. Cross-modal models generate semantically meaningful input representations that have been shown to be highly effective on zero-shot classification tasks (Radford et al., (021)). Cross-modal models that have inspired our work include CLIP for natural images (Radford et al. 2021), COnVIRT for medical images (Zhang et al. 2020), and WikiSatNet (Uzkent et al. 2019) for satellite imagery.
3 SLICE DISCOVERY PRELIMINARIES
In this section, we formulate the slice discovery problem. Consider a standard classification setting with input X E X (e.g. an image, time-series, or graph) and label Y E y' over [)] classes. Addition- ally, assume there exists a set of k slices that partition the data into coherent (potentially overlapping) subgroups, where each subgroup captures a distinct concept or attribute that would be familiar to a domain expert. For each input, we represent slice membership S { example, in the scenario presented in Section [ X represents chest X-rays, Y is a binary label indi- cating the presence of collapsed lungs, and S = (S()) 5(2)) represents two slices: one consisting of normal X-rays with chest drains and the other consisting of collapsed lungs without chest drains.
{0, 1)*,, As an
1==1
The inputs, labels, and slices vary jointly according to a probability distribution P(X, Y) over X y x (0, 1 We assume that training, validation and test data drawn independently and
3

---

Published as a conference paper at ICLR 2022
identically from this distribution. For some upplication-dependenn value of model he X y exhibits degraded performance with respect to slice (D) and metric i y x y (( Y)] < (xx
R if
Assuming that trained classifier he X y exhibits degraded performance on each of the slices in S, we define the slice discovery problem as follows:
Inputs: trained classifier hg and labeled dataset D {(r,, y1) ) with n samples drawn from P(X,Y) Y)
(0, 10, that partition the Output: set of k slicing functions v (1b(e) X V
data into & subgroups.
We consider an output to be successful if, for each ground truth slice S("), slicing function gb(r) predicts 5(() with precision above some threshold B:
Vu E [k]. 30 E [k.. P(g(") 1/bb >>
A slice discovery method (SDM), M (D, V, aims to solve the slice discovery problem.
SLICE DISCOVERY EVALUATION FRAMEWORK
It is challenging to measure how well an SDM satisfies the following desiderata outlined in Section I (a) the model hg should exhibit degraded performance on slices predicted by the SDM and (b) slices predicted by the SDM should be coherent. Most publicly available machine learning datasets do not provide labels identifying coherent, underperforming slices. As result, existing evaluations are either qualitative Eon al., 202T). synthetic (Yeh al. 2020b), or small-scale (Sohoni al., 2020). In this section, we propose an evaluation framework for estimating the success rate of an SDM: how often it successfully identifies the coherent slices on which the model underperforms.
We propose evaluating SDMs across large sets of slice discovery settings, each consisting of: (1) a labeled dataset D {<),,,)) }/ (2) a model hg trained on D, and (3) ground truth slice annotations {s:) ) for one more coherent slices S which the model he exhibits degraded performance, As discussed Section (1) and (2) correspond to the inputs to the SDM, while (3) corresponds to the expected output. Using these slice discovery settings, we can estimate how often an SDM (D, fo) + successfully identifies the slices S. Algorithm ] details the procedure.
Algorithm SDM Evaluation Process
for (D,s. ha) E settings do
V f- M (Dvalid ho) > Fit the SDM on the validation set, yielding set of slicing functions W
for j E [k] do
g(5) + (0)) (Drea)
D Apply the slicing functions to the test set, yielding $ E [0, 1] 1|=
end for
metrics + max maxxelk jElk)
((s()) {
Compute metric L comparing $ and
end for
In Section 4. propose process for generating representative slice discovery settings and in Section we describe the evaluation metrics L and datasets that we use.
4.1 GENERATING SLICE DISCOVERY SETTINGS
To obtain accurate estimates of SDM performance, our slice discovery settings should be both rep- resentative of real-world slice discovery settings and large in number. However, such slice discovery settings aren't 't readily available, since few datasets specify slices on which models underperform.
Here, propose framework for programmatically generating a large number of realistic slice discovery settings. We begin with a base dataset Drase that has either a hierarchical label structure (e.g. ImageNet) or rich metadata accompanying each example (e.g.. CelebA). We select target
4

---

Published as a conference paper at ICLR 2022
variable Y and slice variable S, each defined in terms of the class structure metadata. This allows us to derive target and slice labels (.))2 directly from the dataset. In addition, because the slice S is defined in terms of meaningful annotations, we know that the slice is coherent. After selecting a target variable and slice variable, we (1) generate a dataset 'D (Section 4.1.1 and (2)
generate model hg that exhibits degraded performance with respect to S (Section 4.1.2
4.1.1 DATASET GENERATION
We categorize each slice discovery setting based on the underlying reason that the model hg exhibits degraded performance on the slices S. We survey the literature for examples of underperforming slices in the wild, which we document in Section A.2 Based on our survey and prior work (Oakden- Rayner et al. 2019), we identify three common slice types: rure slices, correlation slices, and noisy label slices. We provide expanded descriptions in Section/A.3
Rare slice. Models often underperform on rare data slices, which consist of data subelasses that occur infrequently in the training set (e.g. patients with rare diseases, photos taken at night). To generate settings with rare slices, we construct D such that for a given class label Y, elements in 0.1.
subclass C' occur with proportion a, where 0.01
Correlation slice. If a target variable Y (e.g. pneumothorax) is correlated with another variable C (e.g. chest tubes), the model may learn to rely on C. to make predictions. To generate settings with correlation slices, we construct such that a linear correlation of strength a exists between the target variable and other class labels, where 0.2 a < 0.8. (Details in Section (A3..)..
Noisy label slice. Particular slices of data may exhibit higher label error rates than the rest of the training distribution. To generate settings with noisy labels, we construct D such that for each class label Y, the elements in subclass C' exhibit label noise with probability a, where 0.01 < a < 0.3.
4.1.2 MODEL GENERATION
Each slice discovery setting includes a model hg that exhibits degraded performance with respect to set of slices S. We propose generating two classes of models: (a) trained models, which are realistic, and (b) synthetic models, which provide greater control over the evaluation.
Trained Models. We train a distinct model he across each of our generated datasets D. If the model he exhibits a degradation in performance with respect to the slices S, then the model hg. dataset D, and slices S will comprise valid slice discovery setting. (See Section 3 for the formal definition of degraded performance; we use accuracy for our metric é and set e 10.)
Synthetic Models. Because the trained model he may underperform S. an SDM that fails to identify may still recover other coherent, underperforming slices. This complicates the interpretation of our results. To address this subtle issue, we also create settings using synthetic models h [0, 1] x y + [0, 1] that simulate predictions. This allows us to distribute errors outside of randomly, ensuring that there are likely no underperforming, coherent slices outside of S. We simulate predictions by sampling from beta distributions (see Section A.3.4).
coherent slices other than
4.2 EVALUATION APPROACH
We instantiate our evaluation framework by generating 1,235 slice discovery settings across a num- ber of different tasks, applications, and base datasets. Detailed statistics our settings are provided We generate slice discovery settings the following three domains:
in Table [
Natural Images (CelebA and ImageNet): The CelebFaces Attributes Dataset (CelebA) includes over 200k images with 40 labeled attributes (Liu et al. 2015). ImageNet includes 1.2 million images across 1000 labeled classes organized in a hierarchical structure (Deng et al. 2009 Fellbaum 1998).
Medical Images (MIMIC-CXR) The MIMIC Chest X-Ray (MIMIC-CXR) dataset includes 377,110 chest x-rays collected from the Beth Israel Deaconess Medical Center. Annotations in- dicate the presence or absence of fourteen conditions (Johnson et al. 2019 2020).
Medical Time-Series Data (EEG): In addition to image modalities, also explore seress data. We obtain a dataset of short 12 second elettroencephalography (EEG) signals, which have been used in prior work for predicting the onset of seizures (Saab al. 2020).
5

---

Published as a conference paper at ICLR 2022
Base Dataset
A Ag A
Slice Discovery Setting
Setting Specification
Sampled Datesct D
1
1
Type + Correlation
Y +- Ai
0
1
Pracsion-at-
α
S - A 2
0
0.4
1 1
Setting Butlder
hgg
Precisionatt10
Figure 2:
Evaluation Framework. We propose framework for generating slice discovery settings from any base dataset with class structure metadata.
We evaluate SDM performance with precision-at-k, which measures the proportion of the top k elements in the discovered slice that are the ground truth slice. We use k: 10 this work.
DOMINO
In this section, we introduce Domino, an SDM that uses cross-modal embeddings identify coher- ent slices and generate natural language descriptions. Domino follows three-step procedure:
Embed (Section 5.1 We encode the inputs {=:) cross-modal embedding space via function ginput X
R& We learn this embedding function Fiopur jointly with embedding function 9text T RRd that embeds text in the same space the inputs.
2. Slice (Section 5.2): We identify underperforming regions in the cross-modal embedding space using an error-aware mixture model fit the input embeddings Zippur {z Pinpur (x;)) model predictions {úr = ho [=1 and true class labels (ys | This yields k slicing functions of the form v/ (j)
slice
X
1).
3. Describe (Section
Finally, we use the text embedding function Itex learned in step (1) to generate a set of & natural language descriptions of the discovered slices.
5.3
5.1 EMBEDDING INPUTS WITH CROSS-MODAL REPRESENTATIONS
Cross-modal representation learning algorithms embed input examples and paired text data in the same latent representation space. Formally, given a dataset of paired inputs V E X and text descrip- tions T, Ppaired = {(v,, 2mm Ppaarrdd we learn two embedding functions Jinpot X + Rd and Stext : T - Rdd such that the distances between pairs of embeddings distt(gnputtee Prex (txx(tt))) (t₃)) reflect the semantic similarity between it and t, for all i, S Apaired.
Ultimately, this joint training procedure enables the creation of semantically meaningful input em- beddings that incorporate information from text. In this work, key insight that input represen- tations generated from cross-modal learning techniques encode the semantic knowledge necessary for identifying coherent slices. Our method relies the assumption that have access to either (a) pretrained cross-modal embedding functions or (b) a dataset with paired input-text data that can be used learn cross-modd. embedding functions. It is important note that paired data is only required if practitioner wishes to generate custom cross cross-modal embedding functions.
Domino uses four types of cross-modal embeddings enable slice discovery across our input do- mains: CLIP (Radford et al. 2021), CIRVIRT (Zhang et al., 2020). MIMIC-CLIP and EEG-CLIP. We adapt CLIP and VIRT from prior work, and we train MIMIC CLIP and EEG-CLIP on large datasets with paired inputs and text (implementation details are provided in Section A.4.
5.2 CLUSTERING EMBEDDINGS WITH ERROR-AWARE MIXTURE MODEL
We then proceed to the second step in the Domino pipeline: slicing. Recall that our goal to find a set of slicing functions that partition our data into coherent and underperforming slices. Taking inspiration from the recently developed Spotlight algorithm (d' et al. (022). propose

---

Published as a conference paper at ICLR 2022
mixture model that jointly models the input embeddings, class labels, and model predictions. This encourages slices that are homogeneous with respect to error type (e.g. all false positives). The model assumes that data is generated according to the following generative process: each example is randomly assigned membership to a single slice according to a categorical distribution S ~ Cal(ps)
with parameter ps {p R Σx Ei=P: P: 1}. Given membership in slice j. the embeddings are normally distributed Z Z|5() = = (n O with parameters mean μ I and covariance E $4 (the set of symmetric positive definite d x matrices), the labels vary as a categorical Y|S = ~ Cat(p(o) with parameter p E {p E R& -i=IP: = 1}, and the model Cal(f(l) with parameter p() E {ps _ Σi= P: = 1}. This assumes that the embedding, label, and prediction are all independent conditioned on the slice.
predictions also vary a categorical YISU)
The mixture model is parameterized by ø = [ps, (u y6), , "6 The log-likelihood over the validation dataset D, is given as follows and maximized using expectation-maximizaioon
(6)) log
(1)
where 7 is hyperparameter that balances the trade-off between coherence and under-performanee our two desiderata (see Section A.4.2).
5.3 GENERATING NATURAL LANGUAGE DESCRIPTIONS OF DISCOVERED SLICES
Domino can produce natural language statements that describe characteristics shared between ex- amples in the discovered slices. To generate slice descriptions, we begin by sourcing corpus of candidate natural language phrases Diett = {t, Tlece [t Critically, this text data can be sourced inde- pendently and does not need to be paired with examples in D. In Section A.4.3 describe an approach for generating a corpus of phrases relevant to the domain.
We generate embedding for each phrase in Dees using the cross-modal embedding func- tion Iex. yielding {2)ext := (t, )=s Then, we compute a prototype embedding for each of the discovered slices by taking the weighted average of the input embeddings in the slice, { = (i) (x, y) Zinpur)i=n We also compute prototype embedding for each class 1 E. [8r cl2, J Je=1- To distill the slice prototypes, we subtract the proto- ( the
- type the most common class in the slice < =elnss Finally, to find text that describes each slice, we compute the dot product between the distilled slice prototypes and the text embeddings
and return the phrase with the highest value: argmax jE(mmon 2ext -
=(()
*class
6 EXPERIMENTS
We use the evaluation framework developed in Section to systematically assess Domino, compar- ing to existing SDMs across 1,235 slice discovery settings. Our experiments validate the three core design choices behind Domino: (1) the use of cross-modal embeddings, (2) the use of novel error-aware mixture model, and (3) the generation natural language descriptions for slices. We provide SDM implementation details in Section A.3.2 and extended evaluations in Section A.5
6.1 CROSS-MODAL EMBEDDINGS IMPROVE SDM PERFORMANCE
In this section, evaluate the effect of embedding type on performance. We hold error-aware slicing algorithm (Step 2) constant and vary our choice of embedding (Step 1).
Natural Images. We compare four embeddings: final-layer activations of candomly-initialzze ResNet-50 (He et al. 2016). final-layer activations of hg. BiT (Kolesnikov et al. 2019), and CLIP (Radford et al. 2021 CLIP embeddings are cross-modal. Results shown in Figure
7

---

Published as a conference paper at ICLR 2022
Synthetic Model
Trained Model
Netural Images
100
semeial
le. Jakie
Prediionat:10
66
Fialyy sre
Precason-att-1
Rondom Initaiizatiion
BST (Uvi-moda
Activattons (1ii Mosial
CLIP [Cross Modal
Medical Images
0.0
of sertings
07%
Precsion-4t:10
0:5
6.
106
roley
Prechson-at-10
Imagenet (Unimodad
BIT Mlsimocia)
Activations (Uné Modal
MIMICCUP (Cros-MModa)
ConVitt (Crose Modal
Medical Time-
Precilen-a-10
1.
Preditonat110
Predsion a2-10
EDG (Uni-modal)
EEO-CLI (Croes-Macal)
Figure 3: Cross-modal embeddings enable accurate slice discovery. Using our evaluation frame- work, demonstrate that the of cross-modal embeddings leads to consistent improvements in slice discovery across three datasets and two input modalities (1,235 settings).
When evaluating with synthetic models, we find that using CLIP embeddings results in mean precision-at of 0,570 (95% CI: 0.554, 0.586), percentage point increase over BiT embeddings and 23 percentage point increase over random activations
When evaluating with trained models, we find no difference between using CLIP embeddings and BiT embeddings. However, both outperform activations of the trained classifier hg by nearly 15 percentage points in mean precision-at
This finding is particular interest given that classifier activations are popular embedding choice in prior SDMs (d'Eon et al.) 2021: Sohoni al. 2020). Notably, the gap between CLIP and he activations is much smaller in settings with correlation slices. This makes sense because model that relies correlate to make predictions will likely capture information about the correlate its activations (Sohoni et al. 2020).
Medical Images. We compare five embeddings: the final-layer activations of ResNet-50 pre- trained on ImageNet (He et al., 2016). the final-layer activations of the trained classifier hg, BiT Kolesnikov al.) 20T9), and domain-specific cross-modal embeddings that trained using two different methods: MIMIC-CLIP and Con VIRT. For synthetic models, cross-modal CIRVIRT em- beddings enable mean precision-at-10 of 0.765 (95% CI: 0.747, 0.784), 7 point improvement over the best unimodal embeddings (BiT) with mean precision-ate -atisioon-tt-iie 10 of 0.695 (95% CI: 0.674, 0.716). For trained models, again find that although he activations perform the worst on rare and noisy label slices, they are competitive with cross-modal embeddings on correlation slices.
Medical Time Series. For our EEG dataset, we compare the final-layer activations of pretrained seizure classifier and CLIP-style cross-modal embedding trained on EEG-report pairs. When evaluating with synthetic models, we find that cross-modal embeddings recover coherent slices with mean precision-at- 10 of 0.697 (95% CI: 0.605, 0.784). This represents 17 point gain over using unimodal embeddings 0.532 (95% Cl: 0.459, 0.608). Cross-modas embeddings also outperform unimodal embeddings when evaluated with trained models. This demonstrates that cross-modal embeddings can aid in recovering coherent slices even in input modalities other than images.
Note that synthetic models do not have activations,
compare trained activations here.
8

---

Published as a conference paper at ICLR 2022
Synthetic Model
Trained Model
000
1300
Natural Images
Precison-ath1
sattingg
750
2000
200
Precsion-tt10 10
Predidon-t110
Confeston Matrix
Mutticouracy
Spotiigtt
GEORGE
Error Micture Model
Figure 4: Error-aware mixture model enables accurate slice discovery. When cross-modal embeddings are provided as input, our error-aware mixture model often outperforms previously- designed SDMs. Results on medical images and medical time-series data are in Section A.5
6.2 ERROR-AWARE MIXTURE MODEL IMPROVES SDM PERFORMANCE
In order understand the effect of slicing algorithms on performance, we hold the cross-modal embeddings (Step 1) constant and vary the slicing algorithm (Step 2). We compare the error- aware mixture model four prior SDMs: George (Sohoni al. 2020), Multiaccuracy (Kim al. 2018), Spotlight (d Eon et al. 2021), and a baseline we call ConfusionSDM, which outputs slicing func- tions that partition data into the cells of the confusion matrix. We provide rross-modal embeddings as input all five SDMs. On noisy and rare slices in natural images, the error-aware mixture model recovers ground truth slices with a mean precision-at- the next-best method, George. Interestingly, correlation slices, the naive ConfusionSDM baseline outperforms our error-aware mixture model. Extended evaluations are provided in Section A.5
0.639 (95% CI: 0.617,0.66 1,6,0.660) this
represents 105% improvement
6.3 DOMINO GENERATES NATURAL LANGUAGE DESCRIPTIONS OF DISCOVERED SLICES.
Domino is the first SDM that can generate natural language descriptions for identified slices. For natural images, we provide quantitative analysis of these descriptions. Specifically, since Domino returns a ranking over all phrases in the corpus Drex., we can compute the percentage of settings in which the name of the "ground truth" slice (or WordNet synonym (Fellbaum 1998) appears the we plot this percentage for k to k = 10. We find
top-k words returned by Domino. In Figure 9 that for 34.7% of rare slices, 41.0% of correlation slices, and 39.0% of noisy label slices, Domino ranks the name of the slice (or synonym) first out of the thousands of phrases in our corpus. In 57.4%, 55.4%, and 48.7% of rare, correlation, and noisy label slices respectively, Domino ranks phrase in the top ten. In Section A.T. we show that Domino is successful at recovering accurate explanations for natural image, medical image, and medical time series data.
7 CONCLUSION
In this work, we analyze the slice discovery problem. First, we observe that existing approaches for evaluating SDM performance do not allow for large-scale, quantitative evaluations. We address this challenge by introducing programmable framework to measure SDM performance across two axes: underperformance and coherence. Second, we propose Domino, a novel SDM that combines cross-modal representations with an error-aware mixture model. Using our evaluation framework, we demonstrate that the embedding and slicing steps Domino outperform those of existing SDMs. We also show for the first time that using cross-modal embeddings for slice discovery can enable the generation of semantically meaningful slice descriptions. Notably, Domino requires only black- box access to models and can thus be broadly useful in settings where users have API access to models. Future directions include executing controlled user studies to evaluate when generated explanations are actionable, developing strategies for computing input embeddings when access to both pre-trained cross-modal embeddings and paired input-text data is limited, and exploring strategies for improving slice discovery in settings where slice strength, is low. We hope that our evaluation framework accelerates the development slice discovery methods and that Domino will help practitioners better evaluate their models.

---

Published as a conference paper at ICLR 2022
8 REPRODUCIBILITY STATEMENT
We provide an open-source implementation of our evaluation framework at bhttps: / com/ com/Hazyll cww.eaaaeeeeeeccceeaaaaaaaaaaaaa coomiinon Users can run Domino on their own models and datasets by installing our Python package via: pip install domino.
ETHICS STATEMENT
Domino is tool for identifying systematic model errors. No matter how effective it is at this task, there may still be failure-modes Domino will not catch. There is a legitimate concern that model debugging tools like Domino could give practitioners false sense of security, when in fact their models are failing on important slices not recovered by Domino. It is critical that practitioners still run standard evaluations on accurately-labeled, representative test in addition to using Domino for auditing models. Additionally, because Domino uses embeddings trained on image-text pairs sourced from the web, may reflect societal biases when identifying and describing slices. Future work should explore the impacts of using biased embeddings to identify errors in models. What kinds of error modes might we miss? Are certain underrepresented groups or concepts less likely to be identified as an underperforming slice?
10 CONTRIBUTIONS
S.E., M.V., K.S., J.D., J.Z., and C.R. conceptualized the overall study. S.E., M.V., K.S., J.-B.D., J.D., J.Z., and C.R. contributed to the experimental design while S.E., M. K.S., and J.-B.D. wrote computer code and performed experiments. S.E. trained all models for CelebA and ImageNet, M.V. trained all models for MIMIC, K.S. trained all models for EEG, and J.-B.D generated CIRT embeddings for MIMIC. C.L.-M. provided the labeled EEG data and clinical expertise. S.E., M.V., and K.S. prepared the manuscript. All authors contributed manuscript review.
11 ACKNOWLEDGEMENT
We are thankful to Karan Goel, Laurel Orr, Michael Zhang, Sarah Hooper, Neel Guha, Megan Leszezynski, Arjun Desai, Priya Mishra, Simran Arora, Jure Leskovee, Zach Izzo, Nimit Sohoni, and Weixin Liang for helpful discussions and feedback. Sabri Eyuboglu is supported by the Na- tional Science Foundation Graduate Research Fellowship. Maya Varma is supported by graduate Scholars
fellowship awards from the Department of Defense (NDSEG) and the Knight- program Stanford University. Khaled Saab is supported by the Stanford Interdisciplinary Gradu- ate Fellowship with the Wu Tsai Neurosciences Institute. James Zou is supported by NSF CAREER 1942926. We gratefully acknowledge the support of NIH under No. U54EB020405 (Mobilize), NSF under Nos. CCF1763315 (Beyond Sparsity), CCF1563078 ( (Volume to Velocity), and 1937301 (RTML); ARL under No. W911NF-21-2-0051 (Interactive Human-Al Teaming); ONR under No. N000141712266 (Unifying Weak Supervision); ONR N00014-20 100-1-244 1-2480: Understanding and Ap- plying Non-Euclidean Geometry in Machine Learning; 400014201 2275 (NEPTUNE); NXP, Xilinx, LETI-CEA, Intel, IBM, Microsott, NEC, Toshiba, TSMC, ARM, Hitachi, BASF, Accenture, Eric- sson, Qualcomm, Analog Devices, Google Cloud, Salesforce, Total, the HAI-GCP Cloud Credits for Research program, the Stanford Data Science Initiative (SDSI) and members of the Stanford DAWN project: Facebook, Google, and VMWare. The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright notation thereon. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views, policies, or endorsements, either expressed or implied, of NIH, ONR, or the U.S. Government.
REFERENCES
Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, and Been Kim. Sanity Checks for Saliency Maps. In S Bengio, H Wallach, H Larochelle, K Grauman, N Cesa- Bianchi, and R Gamett Advances in Neural Information Processing Systems 31, pp. 9505- 9515. Curran Associates, Inc., 2018.
Advances
10

---

Published as a conference paper at ICLR 2022
Mareus A Badgeley, John R Zech, Luke makden--Raynes Benjamin Glicksberg, Manway Liu, William Gale, Michael V McConnell, Bethany Percha, Thomas M Snyder, and Joel Dudley. Deep learning predicts hip fracture using confounding patient and healthcare variables. npj Digital Medicine, 2(1):1-10, April 2019.
Atul Bansal, Ravinder Agarwal, and K Sharma. Svm based gender classification using iris images. pp. 425-429, November 2012.
Alceu Bissoto, Michel Fornaciali, Eduardo Valle, and Sandra Avila. (de) constructing bias on skin lesion datasets. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition Workshops, pp. 0-0, 2019.
Joy Buolamwini and Timnit Gebru. Gender shades: Intersectional accuracy disparities in commer- cial gender classification. In Conference on fairnesss, accountability and transparency, pp. 77-91. PMLR, 2018.
Mayee Chen, Karan Goel, Nimit S Sohoni, Fait Poms, Kayvon Fatahalian, and Christopher Re. Mandoline: Model evaluation under distribution shift. In Proceedings of the 38th International Conference Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. httpgg/.......//1333333333
1617-1629. PMLR, 18-24 Jul 2021. URL https: httpps:/proceediiiii ngs. w.pppesse/v339
chen21i. html
Yeounoh Chung, Tim Kraska, Neoklis Polyzotis, Ki Hyun Tae, and Steven Fuijong Whang. Slice finder: Automated data slicing for model validation. In 20/9 IEEE 35th International Conference on Data Engineering (ICDE), pp. 1550-1553, iccexplore.ieee.og
April 2019.
Terrance de Vries, Ishan Misra, Changhan Wang, and Laurens van der Maaten. Does object recog- nition work for everyone? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 52-59, 2019.
Alex DeGrave, Jose Janizek, and Su-In Lee. Al for radiographic COVID-19 detection selects shortcuts over signal. Nature Machine Intelligence, 3(7):610-619) May 2021.
Tean-Benoit Delbrouck, Khaled Saab, Maya Varma, Sabri Eyuboglu, Jared A. Dunnmon, Pierre Chambon, Juan Manuel Zambrano, Akshay Chaudhari, and Curtis P. Langlotz. Vilmedic: framework for research the intersection of vision and language in medical ai. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics: System Demonsira- lions. Association for Computational Linguistics, May 2022.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hier- archical image database. 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp.
June 2009.
Greg d' doE, Jason d' EoE, James R. Wright, and Kevin Leyton- Leyton-Brown. The spotlight: A general method for discovering systematic errors in deep learning models, 2021.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. CoRR, abs/ 810.0880,,, 2018. URL
http:://rxiv> http://wwwwgga////iie/eae
org/ abs /albs//810 04805
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image is worth 16x 16 words: Transformers for image recognition at scale, 2021.
Christiane Fellbaum. WordNet: An Electronic Lexical Database. Bradford Books, 1998.
Jason A Fries, Paroma Varma, Vincent S Chen, Ke Xiao, Heliodoro Tejeda, Priyanka Saha, Jared Dunnmon, Henry Chubb, Shiraz Maskatia, Madalina Fiterau. Scott Delp. Euan Ashley, Christo- pher Ré, and James R Priest Weakly supervised classification aortic valve malformations using unlabeled cardiac MRI sequences. Nat. Commun., 10(1), December 2019.

---

Published as a conference paper at ICLR 2022
Karan Goel, Nazneen Rajani, Jesse Vig, Samson Tan, Jason Wu, Stephan Zheng, Caiming Xiong, Mohit Bansal, and Christopher Ré. Robustness Gym: Unifying the NLP Evaluation Landscape. arXiv:2101. 04840 [cs]. January 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. February 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp- 770-778, 2016.
Yue He, Zheyan Shen, and Peng Cui. Towards non-i..d. image classification: A dataset and base- lines. Pattern Recognit., 110:107383, February 2021.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common ruptions and perturbations. March 2019.
A Johnson, L Bulgarelli, T Pollard, Horn. LA Celi, and R Mark. Mimic-iv (version 1.0), 2020.
Alistair E Johnson, Tom Pollard, Seth Berkowitz, Nathaniel Greenbaum, Matthew Lun- gren, Chih-ying Deng. Roger G Mark, and Steven Horn. Mimic-exr, de-identified publicly available database of chest radiographs with free-text reports. Scientific data, 6(1):1-8, 2019.
Aditya Khosla, Nityananda Jayadevaprakash, Bangpeng Yao, and Li Fei-Fei. Novel dataset for fine- grained image categorization. In First Workshop Fine-Grained Visual Categorization, IEEE Conference on Computer Vision and Pattern Recognition, Colorado Springs, CO, June 2011.
Michael P Kim, Amirata Ghorbani, and James Zou. Multiaccuracy; Black- Box Post-Processing for Fairness in Classification. arXiv: 12317 [cs, stat], August 2018.
Diederik P Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. arXiv:1412.
6980 [cs], January 2017.
Allison Koenecke, Andrew Nam, Emily Lake, Joe Nudell, Minnie Quartey, Zion Mengesha, Connor Toups. John R Rickford, Dan Jurafsky, and Sharad Goel. Racial disparities in automated speech recognition. Proceedings of the National Academy of Sciences, 117(14):7684-769) 2020.
Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsub- ramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, Tony Lee, Etienne David, Ian Stavness, Wei Guo, Berton A Earnshaw, Imran S Haque, Sara Beery, Jure Leskovec, Anshul Kundaje, Emma Pierson, Sergey Levine, Chelsea Finn, and Percy Liang. WILDS: A Benchmark of in-the-Wild Distribution Shifts. arXiv:2012, 0742/ [cs], March 2021.
Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, and Neil Houlsby. Big transfer (bit): General visual representation learning. December 2019.
Andrey Kuehlkamp, Benedict Becker, and Kevin Bowyer. Gender-from-irrss gender-from- mascara? February 2017.
Weixin Liang and James Zou. Metadataset: A dataset datasets for evaluating distribution shifts and training conflicts. In /CML.202/ MIAdata Workshop, 2021.
Ziwei Liu, Ping Luo. Xiaogang Wang. and Xiaoou Tang. Deep learning face attributes in the wild.
In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
Luke Oakden Rayner, Jared Dunnmon, Gustavo Carneiro, and Christopher Ré. Hidden stratification causes clinically meaningful failures in machine learning for medical imaging. September 2019.
Laurel Orr, Megan Leszczynski, Simran Arora, Sen Wu, Neel Guha, Xiao Ling, and Christopher Re. Bootleg: Chasing the tail with self-supervised named entity disambiguation. October 2020.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. February 2021.
12

---

Published as a conference paper at ICLR 2022
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. "why should trust you?": Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD International Conference Knowledge Discovery and Data Mining, KDD '16. pp. 1135-1144, New York, NY, USA, 2016. Association for Computing Machinery. ISBN 9781450342322. doi: 10.1145/293967. 2939778. URL URL htttttttt//:/.t.t..
Subhrajit Roy, Isabell Kiral- KornekK and Stefan Harrer. Chrononet: deep recurrent neural network for abnormal eeg identification. In Conference on Artificial Intelligence in Medicine in Europe, pp. 47-56. Springer, 2019.
Khaled Saab, Jared Dunnmon, Christopher Ré, Daniel Rubin, and Christopher Lee-Messer. Weak supervision an efficient approach for automated seizure detection in electroencephalog raphy. npj Digital Medicine, 2020. URL https: htpp:://www..t http://wwwatteeeeeeeeeeee.e
nature. cww/ com/articles/ www.aaatte.o.e..rrrrraeeo.a
s41 746746-80 020 0264 Otarticle-
Svetlana Sagadeeva and Matthias Boehm. Sliceline: Fast, linear-algebra- slice finding for ml model debugging. In Proceedings of the 2021 International Conference on Management of Data, Association for Computing Machinery, New York, NY, USA, June 2021.
pp. 2290-2299.
Shiori Sagawa, Pang Wei Koh, Tatsunori B Hashimoto, and Percy Liang. Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Gener- alization. arXiv:1911. 08731 [es, stat], April 2020.
Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based local- ization. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), Oct 2017.
Ilya Semenov and Shamsul Arefin. Wikipedia word frequency. https://githhbb com/ "Lyy@Seenov//iikipedee -oreeddiaworr---- word- frequency 2019.
Sahil Singla, Besmira Nushi, Shital Shah, Ece Kamar, and Eric Horvitz. Understanding failures of deep networks via robust feature extraction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12853-12862, 2021.
Akshay Smit, Saahil Jain, Pranav Rajpurkar, Anuj Pareek, Andrew Ng, and Matthew Lungren. Combining automatic labelers and expert annotations for accurate radiology report labeling us- ing BERT. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp- 1500-1519, Online, November 2020. Association for Computational org/
Linguistics. doi: 10. 202 O. emn 1p pp-mappmm-mmm ma 117
URL
Nimit Sohoni, Jared Dunnmon, Geoffrey Angus, Albert Gu, and Christopher Ré. No sub- class left behind: Fine-grained robustness in grained classification problems. In H. Larochelle, M. Ranzato. R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 19339-19352. Curran Asso- ciates, Inc., 2020. URL
wwweeiinnnnnnuunuuneeeeeeee neur ips. wwtp///ppee//////1eeee.eee Paper pdf
Juan E Tapia, Claudio Perez, and Kevin W Bowyer. Gender classification from the same iris code used for recognition. JEEE Trans. Inf. Forensics Secur., 1 1000:1760-77000 August 2016.
Burak Uzkent, Evan Sheehan, Chenlin Meng, Zhongyi Tang. Marshall Burke, David Lobell, and Stefano Ermon. Learning to interpret satellite images using wikipedia. In Proceedings of the Twghty-Eighth International Joint Conference on Artificial Intelligence, 2019.
Maya Varma, Laurel Otr, Sen Wu, Megan Leszezynski, Xiao Ling, and Christopher Ré. Cross- domain data integration for named entity disambiguation in biomedical text. In Findings of the Association for Computational Linguistics: EMNLP 2021, pp. 4566-4575 Punta Cana, Do- minican Republic, November 2021. Association for Computational Linguistics. URL https:
7/acl cocclannhollooo www.thooooooooooo/22221120111 findi
388
13

---

Published as a conference paper at ICLR 2022
Julia K Winkler, Christine Fink, Ferdinand Toberer, Alexander Enk. Teresa Deinlein, Rainer Hofmann- Wellenhff,, Luc Thomas, Aimilios Lallas, Andreas Blum, Wilhelm Stolz, and Holger A Haenssle. Association Between Surgical Skin Markings in Dermoscopic Images and Diagnos- tic Performance Deep Learning Convolutional Neural Network for Melanoma Recognition. JAMA Dermatol., 155(10):1135, October 2019.
Chih- Yeh, Been Kim, Serean Arik, Chun-Liang Li, Tomas Pfister, and Pradeep Raviku- mar. On completeness-awart concept-based explanations in deep neural networks. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 20554-20565. Curran Asso- ciates, Inc., 2020a. URL http:://p///pccceed.ssss neurips cc/paperr2220//LLe/ CCD88FFF763111666262223322223311
pdf
Chih-Kuan Yeh, Been Kim, Sercan O Arik, Chun-Liang Li, Tomas Pfister, and Pradeep Ravikumar. On Completeness-aware Concept-Based Explanations in Deep Neural Networks. arXiv:1910. 07969 [cs, stat]. June 2020b.
John Zech, Mareus Badgeley, Manway Liu, Anthony B Costa, Joseph J Titano, and Eric Karl Oermann. Variable generalization performance of a deep learning model to detect pneumonia in chest radiographs: a cross-sectional study. PLoS medicine, 15(11):e,00668, 2018.
Brian Hu Zhang, Blake Lemoine, and Margaret Mitchell. Mitigating Unwanted Biases with Ad- versarial Learning. Association for the Advancement of Artificial Intelligence (AAAI), January 2018.
Yuhao Zhang, Hang Jiang, Yasuhide Miura, Christopher D. Manning, and Curtis P. Langlotz. Contrastive learning of medical visual representations from paired images and text. CoRR, abs/2010.00747, 2020. URL https: //apiiv////xvv.//// www.aawwww..aaaaaaaaaaaaaoeeeeee wttp://ww...rraaaa/.......
00747
14

---

Published as a conference paper at ICLR 2022
A
APPENDIX
CONTENTS
A.I
Examples Slice Descriptions
16
A.2
Extended Related Work: Survey of Slices in the Wild
18
A.3
Extended Description of Evaluation Framework
19
A.3.1
Extended Description of Slice Categories
19
A.3.2
SDM Implementations
20
A.3.3
Extended Description of Trained Models
21
A.3.4
Extended Description of Synthetic Models
21
A.4
Extended Description of Domino
A.4.1
Extended Description of Cross-Modal Embeddings
21
A.4.2
Extended Description of Error-Aware Mixture Model
23
A.4.3
Generating Corpus of Natural Language Descriptions
24
Extended Results
25
A.5.1
Extended Analysis of Cross-Modal Embeddings
25
A.5.2
Extended Analysis of Error-Aware Mixture Model
25
A.3.3
Extended Analysis of Natural Language Descriptions
26
A.5.4
Future Work
26
A.6
Glossary of Notation
27
15

---

Published as a conference paper at ICLR 2022
A.1 EXAMPLES OF SLICE DESCRIPTIONS
In this section, we provide examples of the natural language slice descriptions generated by Domino. Tiguue&Sinclude Sincludes natural language descriptions and representative photos for discovered slices in the natural image domain, Figure includes natural language descriptions in the medical image domain, and Figure includes natural language descriptions in the medical time-series domain.
vehicle
building
food
submarine
mosque
banana
photo
photo mosque
photo of banana
photo of underway
photo of ifiam
photo of potassium
photo of sub
a photo of tripoli
a photo of primate
photo of warship
photo of caliphate
photo of cel
photo of navy
photo of caliph
photo of anarctica
black hair
wearing earrings
blonde hair smiling
correlation oitegi
correlation aice oregthy
correlation conpery
eyeglasses
brown hair 612
photo of man spectacled
la photo woman with brown photo of women with brown hair
photo man
glasses
a photo girl
a photo of woman smiling warmly
photo of a man with refractive glasses
photo of girl with brown hair a photo of girl with reddish hair
photo of
smile
photo of a guy with tinted glesses
photo of a smiling woman
photo men aviator sunglasses
a photo
redaish
a photo of a girl with brownish hair
building
vehicle
food beverage
organ
nolsy label
noisy label
noisy label
barn
locomotive
a photo of barn
a photo of locomotive
a photo of drink
photo winter
photo way
of coffee
photo of snow
photo of rail road
photo of beverage
photo of snowfoll
photo
photo
photo formhouse
photo electriiiation
photo of drinking
Figure 5: Domino produces natural language descriptions discovered slices. Natural language descriptions for discovered slices in (top row) 3 settings randomly selected from the set of the 85 rare slice, natural image settings where Domino includes the exact name of the slice in its top slice descriptions; (middle row) 3 settings randomly selected from the set of the 45 correlation slice, natural image settings where Domino includes exact name of the slice in its top slice descriptions and precision-at-25 exceeds 0.8; and (bottom row) 3 settings randomly selected from the set the 95 noisy label slice, natural image settings where Domino includes the exact name of the slice in its top slice descriptions and precision-at-25 exceeds 0.8. The length of the bars beneath each description are proportional to the dot product score for the description (see Section 5.3 Also shown the top 3-4 images that Domino associates with the discovered slice.
16

---

Published as a conference paper at ICLR 2022
Synthetic Model
Trained Model
cardiac condition
pleural condition
saget
Sige:
sardiomegaly
pneumothorax
alct
mild cardiomegaly...
...there no....pneeee no..ppremmothra....
heart is milcly enlarged...
...there is no focal consolidat on or pnoumothorax...
heart remains mildly enlarged...
there is no... effusionnn or pneumothorax...
heart size remains moderately enlarged...
deneeee preumothorax..
... the heart mi Idly enlarged...
no pleural effusion or pneumothorax..
pleural effusion
lung opacity
correlation
correlation
support devices
pleural effusion
dielysis term
...impoviing pleural effusion on right...
eeneeticc tube again notec...
basal pleural effusions noted...
steemoromy are noted....
...small pleural effusions are persistent...
Gual load pacemaker...
...and sosence pleural effusions...
..ffigh-sided port-a-cath is again seen...
...there is no significant interval change...
lung condition
lung condition
nolay label dine eleyynn
noisy label
cillccry
atelectasis
atelectasis
there is trace bibasilar atelectasis
...mild bibasilar atelectasis slightly improved...
overlying atelectasis...
besel atelectasis..
with overlying atelectasis...
..poobbblle bibasilar atelectasis...
.fllt to most likely represent atelectasis...
...milc bibasilar atelectesis...
mild right besiler stelectasis...
lisely reflect of atelectasis...
Figure 6: Domino produces natural language descriptions for discovered slices in medical image dataset. Here, we provide the top five natural language descriptions for discovered slices in (top row) two slice settings, (middle row) correlation slice settings, and (bottom row) two noisy label slice settings. Colored bars represent accurate slice descriptions, and gray bars represent incorrect descriptions. Note that in our medical examples, the vocabulary consists of physician reports in the training set; since, we are unable to provide the full-text reports due to patient privacy concerns, the figure includes relevant fragments of reports. The length of the bars beneath each description are proportional to the dot product score for the description (see Section 5.3)
seizure
seizure
seizure
care
correlation
noisy label
age
age
25 year old healthy man...
15 month old baby...
15 month old baby...
hour video- recording...
intractable epilapey year old
intract ablle epiipsy...
..corellting seizures roted...
girl ex-35 weeker with MH....
ex centem female with se zure...
..taiic clonic movements...
.o.ooic clanic movements...
... y/o ex male with..
a.waakk awwake/aslesp EEG...
seizure listing less than minute...
Figure 7: Domino produces natural language explanations for discovered slices a medical time-series dataset. Natural language descriptions for discovered slices in 3 of EEG slice discovery settings. In all three settings, the model is trained to detect seizures and underperforms on the slice of young patients. The three settings span all three slice categories and in each Domino describes the slice with reports mentioning young age. Note that the description corpus consists of physician reports the training set; however, we are unable to provide the full-text reports due to patient privacy concerns, so the table includes only relevant fragments. The bar lengths beneath each description are proportional to the dot product score for the description (see Section 5.3
17

---

Published as a conference paper at ICLR 2022
A.2 EXTENDED RELATED WORK: SURVEY OF SLICES IN THE WILD
Several recent studies have shown that machine learning models often make systematic errors on critical data slices. In this section, we provide survey of underperforming slices documented in the literature.
Skin Lesion Classification (Correlation Slice): Bissoto et al. (2019) reveal that models trained to classify skin lesion images depend climically irrelevant information due to biases in the training data. Specifically, a computer vision model trained to classify skin lesions performs poorly images of malignant lesions with color charts (i.e. colored bandages), since color charts more commonly appear with benign lesions.
Melanoma Detection (Correlation Slice): study by Winkler et al. (2019) showed that melanoma detection models trained dermascopie image datasets often rely the pres- ence of surgical skin markings when making predictions. Since dermatologists mark sus- picious lesions during clinical practice with a gentian violet surgical skin marker, dermas- copic images will often include skin markings, causing models to learn spurious correla- tions between markings and the presence of melanoma. Models then underperform on the slice of lesions without markings.
Models trained to detect the presence of pneumothorax (collapsed lungs) have been found rely on the presence of chest drains, device used during treatment (Oakden-Rayner et al. 2019).
Pneumothorax Detection (Correlation Slice):
Hip-Fracture Detection (Rare Slice): Due to the low prevalence of cervical fractures in a pelvic X-ray dataset collected from the Royal Adelaide Hospital, computer vision models trained to detect fractures underperform on this slice (Oakden- et al. 2019).
Hip-Fras Hip-Fracture Detection (Correlation Slice): Badgeley et al. (2019) show that the per- formance of models trained to detect hip fractures from X-rays are sensitive multiple patient-specific and hospital-speciffc attributes. In particular, when the test distribution is subsampled to remove correlations with patient attributes (age, gender, BMI, pain, fall) and hospital attributes (scanner, department, radiation, radiologist name, order time), the model performance drops to close to random.
Gender Classification in Images (Rare Slice): Buolamwini & Gebru (0018) demon- strated that facial analysis datasets are often composed primarily of lighter-skinned subjects (based on Fitzpatrick Skin Types) and as result, three commercial gender classification systems systematically underperform on rare subgroups (e.g. darker faces, female faces).
COVID-19 Detection in Chest X-rays (Correlation Slice): DeGrave et al. (2021) re- veal that some models trained to detect COVID-19 from radiographs do not generalize to datasets from external hospitals, indicating that the models rely on source-specific attributes instead of pathology markers.
Pneumonia Detection in Chest X-rays (Correlation Slice): Zech et al. 2018 evalu- ated pneumonia screening CNNs on three external hospitals, and found that performance on the external hospitals was significantly lower than the original hospital dataset. Addi- tionally, the CNNs were able to very accurately classify the hospital system and depart- ment where the radiographs were acquired, indicating that the CNN features had learned hospital-specific confounding variables.
Weakly Supervised Aortic Valve Malformation Classification (Noisy Label Slice): Weak supervision commonly used in medical machine learning practice to label clin- ical datasets. Using set of labeling functions, |Fies et al.) (2019) train a seakkyysuperrvied model to classify aortic valve malformations. They note that noise in the labels is induced by labeling functions. These may systematically miss coherent slices of data, meaning that a model trained on these labels may underperform on those slices.
Predicting Gender from Photos of the Iris (Correlation Slice): Several studies have reported models capable of predicting person's gender from a photo of their iris Bansal et al. 22122); Tapia et al. (0016). However, Kuchlkamp et al. (2017) show that these models may be relying the presence of mascara make these predictions. This suggests that performance will likely be degraded on the slice of females without mascara and males with mascara.
18

---

Published as a conference paper at ICLR 2022
Speech Recognition (Rare Slice): Koenecke al. (2020)
demonstrated that automated
speech recognition systems have large performance disparities between white and African American speakers. The disparities were traced to the gap in the corpus used to train the model, indicating that African American speakers were a rare slice.
Object Recognition (Rare Slice): A study by Vries et al. (2019) demonstrated that pub- licly available object recognition algorithms often systematically underperform on house- hold items that commonly occur in non-Western countries and low-income communities. This is likely due to objects appearing in different environments as well as differences in object appearance.
Named Entity Disambiguation (Rare Slice): Named entity disambiguation (NED) sys- tems, which map textual mentions to structured entities, play a critical role in automated text parsing pipelines. Several studies have demonstrated that NED systems underperform on rare entities that occur infrequently in the training data (Orr et al. 2020
Varma et al.
1021
A.3 EXTENDED DESCRIPTION OF EVALUATION FRAMEWORK
Domain
Rare Settings
Correlation Settings
Noisy Label Settings
Total settings
Natural
177 (trained)
520 (trained)
287 (trained)
984 (trained)
Images
197 (synthetic)
520 (synthetic)
394 (synthetic)
1. 111 (synthetic)
MIMIC
15 (trained)
176 (trained)
30 (trained)
221 (trained)
55 (synthetic)
352 (synthetic)
55 (synthetic)
462 (synthetic)
EEG
10 (trained)
10 (trained)
10 (trained)
30 (trained)
10 (synthetic)
10 (synthetic)
10 (synthetic)
30 (synthetic)
Table 1: Overview of evaluation framework. We evaluate SDMs across 1,235 (trained) slice discovery settings across three domains, four slice categories, and five slice parameters (a).
A.3.1 EXTENDED DESCRIPTION OF SLICE CATEGORIES
In Section 4.1.1. we describe how we categorize each slice discovery setting based on the underlying reason that the model he exhibits degraded performance the slices S. We survey the literature for examples of underperforming slices in the wild, which we document in Section A.2 Based on survey and prior work (Oakden-Rayner et al. 2019), identify three popular slice types. We provide expanded descriptions below.
Rare slice. Consider slice S {0,1) (e.g. patients with rare disease, photos taken at night) that occurs infrequently in the training set (i.e. P(S ==$=10 1) < a for some small a). Since a rare slice will significantly affect model loss during training, the model may fail to learn classify examples within the slice. To generate settings with rare slices, we use base datasets with hierarchical label schema (e.g. ImageNet (Deng et al. 2009). We construct dataset D such that for a given class label Y. the elements in subclass C occur with proportion a, where a ranges between 0.01 and 0.1.
Correlation slice. If the target variable Y (e.g. pneumothorax) is correlated with another variable C' (e.g. chest tubes), the model may learn to rely on C' to make predictions. This will induce slice 1 + Y] (e.g. pneumothorax without chest tubes and normal with chest tubes) with degraded performance. To generate settings with correlation slices, we use base datasets with metadata annotations (e.g. CelebA (Liu al., 2015)). We sub-sample the base dataset Drase such that the resulting dataset D exhibits linear correlation of strength between the target variable and another metadata label C. Here, a ranges between 0.2 and 0.8.
We now describe our procedure for subsampling the base dataset to achieve desired correlation. Assume we have two binary variables Y, C E {0, 1} (Bernoulli random variables) and a dataset of per. Given target correlation a, would like to subsample the dataset Dowe such that the resulting dataset (yi, c;) ) of size n exhibits a sample correlation between and C of a.
Donse { 188e.
19

---

Published as a conference paper at ICLR 2022
The population correlation between Y and C' is given by a(Y, = cov(Y.C)
and cov(Y,
ay
B| CJ [[Y] For a sample, the unbiased estimator of the covariance
cov(y, c)
" n1 vvs () - -2) n1 Σ
1
Σ
n
BEC cy
Since we know Y and C' are Bernoulli random variables, we can express this in terms of variables like Ry=11c=1 (i.e. the number of samples i where y/: 1 and c; = and Ny=1(i.e the number of samples i where y/=1).
cov c)
12 Ry=11c=1
Bay 17
71
The correlation coefficient can then be expressed in terms of this covariance and the sample standard deviations Sy and sgi
1y=11c=1 Ryy Rom1
(1l 18888
addition to supplying a target correlation a, assume we also have target means μoa and μs (these could be the sample means in the original dataset D for example) and a target sample size Since and C are Bernoulli random variables, we can compute sample standard deviations as 8y = [1 - (a) and se 1b(1 - mb).. We can then derive simple formulas for computing the desired values needed to properly subsample the data:
ny=1 = Ha?l
nc=1 14672
ny=1,c=1 a(n 1)8y8e + 1y=111m11
n
By=1,c=0 = ny=1 - 1y=11c=1
0y=0,c=1 n Ry=1.c=1
1ym00,m00 = 11 (Nym1 + Pomml Mym1.om1)
Noisy label slice. Errors in labels are not always distributed uniformly across the training distri- bution. A slice of data S E {0, 1} may exhibit higher label error rates than the rest of the training distribution. This could be due to number of different factors: classes may be ambiguous (e.g. an- notators labeling sandwiches may disagree whether to include hot dogs), labeling heuristics may fail on important slices (e.g. medical imaging heuristics that only activate on images from one scanner type), or human annotators may lack expertise on certain subsets (e.g. annotators from one country labeling street signs in another). A model he trained these labels will likely exhibit degraded per- formance on S. To generate settings with noisy label slices, we can use a base dataset with metadata annotations (e.g. CelebA (Liu et al. 2015)). We construct dataset D such that for each class label Y, the elements in subclass C' exhibit label noise with probability a, where a ranges between 0.01 and 0.3.
A.3.2 SDM IMPLEMENTATIONS
Spotlight. Eon et al. 2021 search for an underperforming slice by maximizing the expected loss with respect multivariate Gaussian with spherical covariance. We use the authors' implemen- tation provided at htt gi huub com/ gwwgdeon/gngg/e spt.ni/ppptitiiiii We enforce minimum spotlight size equal to 2% the validation data as recommended in the paper. We use a initial learn- ing rate of 10 (the default in the implementation) and apply the same annealing and barriers as the authors. We optimize for 1,000 steps per spotlight (the default in the implementation).
GEORGE. Sohoni et al. 2020) propose identifying underperforming slices by performing class conditional clustering on a U-MAP reduction of the embeddings. We use the implementation pro- vided at wtt.tttps//..m ttttps://iitubae http:///gggttttt..coaoaarr/eeeee
www.eeeeeeaeehhhheee ddein rati i cationn
20

---

Published as a conference paper at ICLR 2022
Multiaccuracy Boost. To identify slices where the model hg is systematically making mistakes, Kim et al. (2018) use ridge regression to learn a function f R - R.+, mapping from an example's embedding 2, E I to the partial derivative of the cross entropy loss with respect to the prediction @l(ho(z;): Ui) 1 (a.))) y/
(2)
ahg
Because this function grows as the absolute value of the residual [ha( - yil grows, a good f should correlate with the residual.
In order to discover multiple distinct slices, the authors repeat this process i times updating the model predictions on each iteration according
lig (++1) (3r4) & e e--nft=) ( (xi),
(3)
where 7) hyperparameter defining the step size for the update.
We use an implementation of Multiaccuracy Boost based on the authors', which was released at https://githum com/
We use 17 0.1 as in the authors' implementation. We fit / on 70% of the validation data and use the remaining 30% for evaluating the correlation with the residual. The authors use the same ratio their implementation.
Confusion SDM. A simple, embedding-agnosse way to identify underperforming subgroups is to simply inspect the cells of the confusion matrix. We include this important baseline to determine when more complicated slice discovery techniques are actually useful.
A.3.3 EXTENDED DESCRIPTION OF TRAINED MODELS
In Section 4.1.2. we discuss training a distinct model he for each slice discovery setting. In this section provide additional details on model training.
For our natural image settings and medical image settings, we used a ResNet-18 randomly ini- tialized with He initialization (He et al. 2015 2016). We applied an Adam optimizer with learning rate 1 x 10-4 for 10 epochs and use early stopping using the validation dataset (Kingma & Ba 2017). During training. we randomly crop each image, resize to 224 224, apply random hori- zontal flip, and normalize using ImageNet mean and standard deviation (μ² = [0.485, 0.456, 0.406). [0.229, 0.224, 0.225)). During inference, we resize to 256 x 256, apply center crop of size 224 x 224 and normalize with the same mean and standard deviation as in training.
For our medical time series settings, we use densely connected inception convolution neural network (Roy et al.) 2019) randomly initialized with He initialization (He et al., 2015 2016). Since the EEG signals are sampled at 200 Hz, and the EEG clip length 12 seconds, with 19 EEG electrodes, the input EEG has shape 19 2400. The models are trained with a learning rate of 10-6 and batch size of 16 for 15 epochs.
A.3.4 EXTENDED DESCRIPTION OF SYNTHETIC MODELS
In Section 4.1.2 discuss how synthesizing model predictions in order to provide greater control over the evaluation. Here, we provide additional details describing this process.
Assume binary class label Y {0, 1) and slice variable S E {0, 1). We sample the predicted probability Y e [0, 1] from one of four beta distributions, conditional on and S. The parameters of those four beta distributions are set so as to satisfy desired specificity and sensitivity in the slice 0).
(i.e. when 1) and out of the slice (i.e. when
For natural image settings, we set both specificity and sensitivity to 0.4 in the slice and 0.75 out of the slice. For medical image and time series settings, we set both specificity and sensitivity to 0.4 in the slice and 0.8 out of the slice.
A.4 EXTENDED DESCRIPTION OF DOMINO
A.4.1 EXTENDED DESCRIPTION OF CROSS- EMBEDDINGS
Here, we provide implementation details for the four cross- cross-modes modal embeddings used in this work:
CLIP, CIRT MIMIC- and EEG-CLIP. Domino relies the assumption that have
21

---

Published as a conference paper at ICLR 2022
cess to either (a) pretrained cross- moosl embedding functions or (b) dataset with paired input-text data that can be used to learn embedding functions.
Large-scale pretrained cross-modal embedding functions can be used to generate accurate represen- tations of input examples. For instance, if our inference dataset consists of natural images, we can use pre-trained CLIP model embedding functions Pimpui and Itex to obtain image embeddings that lie in the same latent representation space as word embeddings.
However, trained cross- modsl embeddings are only useful if the generated representations accu- rately represent the inputs in the inference dataset. For example, if the inference dataset consists of images from specialized domains (i.e. x-rays) or non-image inputs, CLIP is likely generate poor representations.
If pre-trained cross-modal embeddings are not available or cannot effectively represent the inference dataset, we require access to separate dataset that can be used to learn the cross-modal embedding functions Finput and Muxi we will refer to this dataset as the CM- Crrrnnniig dataset for the remainder of this work. The CM-training dataset must consist of paired input-text data (e.g. image-caption pairs or radiograph-report pairs). Further, we assume that the text data provides sufficient description of the input and includes information about potential correlates, such as object attributes or subject demographics. Note that paired input-text data is only required for the CM-training dataset; we make no such assumptions about the slice discovery dataset.
We use the following four cross-modal embeddings in our analysis:
CLIP (Natural Images): CLIP embeddings are cross-modal representations generated from a large neural network trained on 400 million image-text pairs (Radford et al.) (021).
ConVIRT (Medical Images): Con VIRT embeddings are generated from pairs of chest X- rays and radiologist reports in the MIMIC-CXR dataset. We create a CM-training set with 70% of the subjects in MIMIC-XRR, ensuring that no examples in our training set occur in validation or test data at slice discovery time. We then replicate the training procedure detailed in Zhang et al. (020)) which uses contrastive learning to align embeddings. We use the implementation provided in ViLMedic (Delbrouck et al. 2022).
MIMIC (Medical Images): We generate a separate set of cross-modal embeddings for the MIMIC-CXR dataset using variant of the CLIP training procedure (Radford al.. 2021) We use the same CM-training set as detailed above, with 89,651 image and report pairs. In order to generate text representations, we extract the findings and impressions sections from radiologist reports, which then serve as input to a BERT- based transformer initialized with weights from Che and then frozen (Smit al. 2020). Chest x-rays are passed as input to a visual transformer (VIT) pre-trained on Net-2kk and ImageNet 2012 2 at an image resolution of 224 x 224 (Dosovitskiy et al. 2021). ) All images are resized to 224 224 and normalized using mean and standard deviation values from ImageNet. Text representations are extracted from the output of the [CLS] token and image representations are extracted from the output of the final model layer. Image representation and text representations are separately passed through projection layers consisting of fully- aully-connected layers and non- linear activation functions detailed in open-source implementation of CLIP" Finally, we align text and image representations using the InfoNCE loss function with in-batch negatives as detailed in Radford al. (202T). We train our implementation for 30 epochs with a learning rate of 10 ", a batch size of 64, and an embedding dimension of 256. The training process comes an early stop if the loss fails to decrease for ten epochs.
EEG (Medical Time-Series Data): order to generate cross-modal embeddings for EEG readings and associated neurologist reports, we modify the CLIP training procedure to work with time-series data (Radford et al., [202 [aab et al., 2020). We create CM-training set with 6,254 EEG signals and associated neurologist reports. We use the same EEG order to represent text representations, we extract the findings and narrative from the reports, which are then served as input to BERT- based transformer initialized with weights from CheXBert (Smit et al., 2020). Text and EEG representations are aligned using the InfoNCE loss function with in- in-batch negatives as described in (Radford et al. 2021). We also add binary cross-entropy classification loss
encoder described in Section AA.33]
22

---

Additional implementation details. The mixture model's objective encourages both slices with high concentration of mistakes as well as slices with a high concentration of correct predictions. However, in the slice discovery problem described in Section 3 the goal is only to identify slices of mistakes (i.e. slices that exhibit degraded performance with respect to some model h2). To reconcile the model's objective with the goal of slice discovery, we model >> slices and then select slices Published as a conference paper at ICLR 2022
for seizure classification, where weigh the InfoNCE loss by 0.9, and the cross-entropy loss by 0.1. The cross-mode model is trained with a learning rate of 10 an embedding dimension of 128, and batch size of 32 for 200 epochs.
A.4.2 EXTENDED DESCRIPTION OF ERROR-AWARE MIXTURE MODEL
In Section 5.2 we describe a mixture model that jointly models the input embeddings, class labels, and model predictions. Here, we provide an expanded description of the model and additional implementation details.
Motivation and relation to prior work. Recall from Section that our goal is to find set of slicing functions that partition our data into subgroups. This task resembles standard unsupervised clustering problem but differs in important way: are specifically interested in finding clus- ters where the model makes systematic prediction errors. It is not immediately obvious how this constraint can be incorporated into out-of-the- the-box unsupervised clustering methods, such princi- pal component analysis or Gaussian mixture models. One potential approach would be to calculate the model Ioss each example x; in D.. append each loss value to the corresponding embedding (T;), and cluster with standard methods (Sohoni et al. 2020). Empirically, we find that this approach often fails identify underperforming slices, likely because the loss element is drowned out by the other dimensions of the embedding.
Recently, Eon al. (2021) proposed Spotlight, an algorithm that searches the embedding space for contiguous regions with high-loss. Our error-aware mixture model is inspired by Spotlight, but differs in several important ways:
Our model partitions the entire space, finding both high and low performing slices. In contrast, Sporligh searches only for regions with high loss. Spotligtt introduces the "spot- light size" hyperparameter, which lower bounds the number of examples in the slice and prevents Spotlight from identifying very small regions.
Because we model both the class labels and predictions directly, error-aware mixture model tends to discover slices that are homogeneous with respect to error type. On the other hand, Sporlight's 's objective is based the loss, function of the labels and predictions that makes false positives and false negatives indistinguishable (assuming cross-entropy).
Eon et al. (202I) recommend using a spherical covariance matrix, Σ al with a E R, when using Sporlight because it made their Adam optimization much faster and seemed to produce good results. In contrast, we use a diagonal covariance matrix of the form Σ diag(a1, (0ng(a11000 (2)......) Fitting our mixture model with expectation maximization remains tractable even with these more flexible parameters.
with the highest concentrations of mistakes. Specifically, model l 25 slices and return the top k clusters with the largest absolute difference between P and p. >> Ip. -p)).
In practice, when d is large (e.g. > 256), we first reduce the dimensionality of the embedding to d 128 using principal component analysis, which speeds up the optimization procedure signifi- cantly.
We include an important hyperparameter ? E R4 that balances the importance of modeling the class labels and predictions against the importance of modeling the embedding. The log- lieliiol over n examples is given as follows and maximized using pupectation-maximiiiii
{(0) log
(4)
In practice, this hyperparameter allows users to balance between the two desiderata highlighted in Section [ When 7 is large (i.e. 7 > 1), the mixture model is more likely to discover underper-
23

---

Published as a conference paper at ICLR 2022
forming slices, potentially at the expense of coherence. On the other hand, when y is small (i.e. S Y < 1), the mixture model is more likely to discover coherent slices, though they may not be underperforming. In our experiments, as they explore model errors with Domino, decreasing it when the discovered slices seem incoherent and increasing it when the discovered slices
set ? 10. We encourage users to tweak this parameter
underperforming.
We initialize our mixture model using a scheme based on the confusion matrix. Typically, the parameters of Gaussian mixture model are initialized with the centroids of k-means fit. However, in our error-aware mixture model, we model not only embeddings (as Gaussians), but also labels and predictions (as categoricals). It is unclear how best combine these variables into single k-means fit. We tried initializing by just applying k-means to the embeddings, but found that this led to slices that were too heterogeneous with respect to error type (e.g. mix of false positives and true positives). Instead, we use initial clusters where almost all of the examples come from the same cell in the confusion matrix. Formally, at initialization, each slice is assigned a yاو E y and g E y (i.e. each slice is assigned cell in the confusion matrix). This typically done in round-robin fashion so that there are at least | [ slices assigned to each cell in the confusion matrix. Then, fill in the initial responsibility matrix Q E gnxh, where each cell Qı corresponds to model's initial estimate of P(SU) 1| yv Y ÿt). We do this according
Qi + {
l y = y A d: = g(()
(5)
otherwise
Qit
Qú + ΣE= Qu
(6)
where is random noise which ensures that slices assigned to the same confusion matrix cell won't have the exact same initialization. We sample c uniformly from the range [0, E| where E hyperparameter set to 0.001.
A.4.3 GENERATING A CORPUS OF NATURAL LANGUAGE DESCRIPTIONS
In Section 5..3 we describe an approach for generating natural language descriptions of discovered slices. This approach uses cross-modal embeddings to retrieve descriptive phrases from large cor- pus of text Diex. In this section, we describe how we curate domain-specifie corpora of descriptions to be used with Domino.
First, we solicit a set of phrase templates from a domain expert. For example, since CelebA is a dataset of celebrity portraits, we use templates like:
a photo of person [MASK] [MASK]
a photo of [MASK] woman
a [MASK] photo of a man
In our experiments with CelebA, we use total of thirty templates similar to these (see GitHub for the full list). In our experiments with ImageNet we use only one template: a photo of [MASK).
Next, we generate large number of candidate phrases by filling in the [MASK] tokens using either (1) a pretrained
languuge model (in our experiments with CelebA we use BERT base model (Devlin et al. 2018) and generate 100,000 phrases, keeping the next 10,000) with the lowest loss) or (2) a programmatic approach (in our experiments with ImageNet, we create one phrase from each of the Thex 10,000 000 most frequently used words English Wikipedia (Semenov & Arefin 2019)).
For our medical image and time-series datasets, MIMIC-CXE (Johnson et al.) 2019) and Saab et al. (2020) with 7 = 159,830 and neext = 41,258, respectively.
corpora of physician reports sourced from
24

---

Published as a conference paper at ICLR 2022
A.5
EXTENDED RESULTS
A.5.1 EXTENDED ANALYSIS OF CROSS-MODAL EMBEDDINGS
explore the effect of embedding type slice discovery performance. Here, provide an extended evaluation of our results.
In Section 6.1
We note that slice discovery performance is generally lower across slices when compared to correlation and noisy label slices. This trend is visible for both model types (synthetic and trained) and all embedding types (unimodal and cross-modal). This is likely due the nature of the rare slice setting; since rare subclass occurs in the dataset with very low frequency, it is difficult for SDMs identify the error slice.
Slice discovery performance synthetic models is often higher than performance on trained mod- els. This expected because the use of synthetic models allows us explicitly control performance on our labeled ground-truth slices, and as result, Domino can more effectively recover the slice. On the other hand, trained models are likely to include underperforming, coherent slices that are not la- beled "ground-truth", which may limit the ability of Domino to recover the labeled slice. Trained models are also likely to exhibit lower slice performance degradations than synthetic models. We discuss these trade-offs in Section 4.2
A.5.2 EXTENDED ANALYSIS ERROR-AWARK MIXTURE MODEL
In Section 6.2
we explore how the choice of slicing algorithm affects slice discovery performance. Here, provide extended evaluation of results.
Additional experimental results with our error-aware mixture model are shown in Figure g
Synthotic Model
Trained Model
3500
Netural Images
Prediion-t-00
0.6
500
Precalonett10
Prediiio-at-11
420
Prediion-er 1o:oe-----10
0e
settings
0.5
0.5
Pracalonat-10
Prechelan-ebb10
Perdilon-tt-0
c1 swrtting
20
0.25 2.5
5.15
1.00
Camerarr
Preckfon-3t 10
Prechion-at-10
Multisscumayy
Spotlight
Figure 8: Error-aware mixture model enables accurate slice discovery. We show that when cross-modal embeddings are provided as input, our error-aware mixture model often outperforms previously-designed SDMs.
We note that the naive Confusion SDM demonstrates high performance on correlation slices across outperforming error-aware mixture model in some cases. This find-
all three datasets, ing suggests that when a strong correlation exists, simply inspecting the confusion matrix may be sufficient for effective slice discovery.
25

---

Published as a conference paper at ICLR 2022
Our error-aware mixture model demonstrates significantly higher performance rare slices than prior SDMs; this is especially visible in trained model results. This is likely because our error-aware mixture model jointly models the input embeddings, class labels, and model predictions, allowing for better identification of rare slices when compared to existing methods.
A.5.3 EXTENDED ANALYSIS NATURAL LANGUAGE DESCRIPTIONS
Please refer to Figure for quantitative evaluation natural image descriptions.
rare
correlation
noisy labe
70
% of settings
60
40
30
Slice mentioned in top-x predictions
Figure 9: Descriptions of discovered slices align with the names of the ground truth slices. Here, show the fraction of natural image settings where Domino includes the exact name of the ground truth slice (or one of its WordNet synonyms (Fellbaum) 1998)) in the top-k slice descriptions.
A.5.4 FUTURE WORK
Based on our analysis of model trends, we identify several directions for future work. First, we observe that slice discovery is particularly difficult when the strength of the slice (a) low. In the future, aim to explore strategies for improving slice discovery and explanation generation in this scenario. Additionally, we hope to explore strategies for generating informative input embeddings when access to paired input-text data limited. Finally, we intend to run controlled user studies in order to understand when explanations generated by Domino are actionable for practitioners and domain experts.
26

---

Published as a conference paper at ICLR 2022
A.6 GLOSSARY OF NOTATION
Classification (Section 3
X The set of values that the inputs can take on a standard classification setting. For exam- ple, this could be the set of all possible 256 x 256 RGB images.
y The of values that the labels can take on in standard classification setting. this work, we deal primarily with binary classification where y = {0, 1}.
X A random variable representing the input in standard classification setting.
Y A random variable representing the label in a standard classification setting.
P A probability distribution. For example, the joint distribution over inputs and labels can be expressed as P(X,Y)
Z4 The realization of the 8 sample of X.
yi The realization of the 7th sample of Y.
n The number of samples in classification dataset.
x The set of n samples X, such that x = [=x)=== E 2
The set of n samples of Y, such that y (1i) E )m
D labeled dataset sampled from P(X, Y). such that D {x y;)}
hg A classifier with parameters 8 that predicts from X, where he X + y.
Y A random variable representing the prediction of the model, such that Ý = ha (X).
L A performance metric for standard classification setting, where l y R (e.g.
accuracy).
Slice Discovery (Section
(G)
A random variable representing the j data slice. This is binary variable SU) {0, 1}. The realization the 7th sample S(i)
(
k:
The number of slices in the data.
S
A random variable representing a set of k slices S = {86)) E 1).
The realization the 7th sample of S.
s.
60
A slicing function أهاي X' x y - 10
k-
The number of slicing functions returned by a slice discovery method.
V
A set of k slicing functions V (1) 1=1-
M A slice discovery method, MD, + V.
Evaluation Framework (Section 4
L A slice discovery metric, L 0, 1 [0. + E (e.g. precision-at-10) 10).
C A random variable representing metadata attribute. We use C' to generate datasets with underperforming slices.
O The strength of generated slice. For example, in correlation slice, a is the Pearson correlation coefficient between the label Y and the correlate C.
6 The performance degradation in metric required for a model to be considered underper- forming.
hgg
A synthetic model, : [0, 1/ x y - [o, 1]. which samples predictions Ý conditional on
V and S.
27

---

Published as a conference paper at ICLR 2022
Domino (Section 5
T
The set of possible text strings.
random variable representing a text string in a paired data setting, T E
A random variable representing imputs in paired data setting, V E A. Note that we do use here in order emphasize the difference between the data used train the classifier and the data used to learn cross-modal embeddings.
pairee
The number of examples paired dataset.
D paired
44) where the text l describes the input u, The dimensionality of the embeddings.
paired dataset {(v,, 4.) Pruind
An embedding function for inputs, Finpur X - Rd
Jimpat
Impat
An embedding function for text, flext
Rdd
Z
A random variable representing the embedding of an input, such that Z -inpur (X).
The value of the 7t sample of Z, such that 2; Puepui (I;).
The set of samples of Z, such that z { E Raxd
Zest
random variable representing the embedding of a text string, such that Z = Otexs (T)).
text
The realization of the 7th sample of Zuxt such that 2 Shex (l)x((44.
z(()
The average embedding of the jth slice.
Ssicce
(()
The average embedding the 5th class.
class
10
In the error-aware mixture model, the mean parameter of the Gaussian distribution used to model Z for slice.
0
In the error-aware mixture model, the covariance parameter of the Gaussian distribution used model Z for the 7th slice.
p
In the error-aware mixture model, the parameter of the categorical distribution used to model y for the 8th slice.
((0) In the error-aware mixture model, the parameter of the categorical distribution used to model y for the th slice.
The parameters of the error aware mixture model: ø = [Pss {ult), E(s), p()), (()))
28


Reviews:
[{"review_id": "n6_Ebz3o0xW", "reviewer": "Reviewer_LuSN", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper addresses interesting problems in the area of slice discovery, particularly underperforming clusters. Overall the Domino approach seems to work well for this task. There are ablation studies on the embeddings used and clustering algorithm chosen. The part about generating natural language explanation is not as convincing as it's ultimately using the text embeddings on a large corpus in a retrieval setting. Actually generating text from the given slice embeddings would have been more convincing. Currently the text seems to be restricted to single words from Fig 5. \nThough the idea introduced in the paper is quite interesting, the paper itself is organised in a very confusing manner. Related work is only introduced in section 6 on page 9 and seems incomplete. The Domino method is shown in Figure 1 on page 2 but the actual text describing it is on page 6. Another weakness is the novelty. Though the framework is novel, the individual models are not. Given the new evaluation framework, it would have been great to introduce some further technical novelty. \nIn the experiments section, the results are described as-is without further interpretation. Do the results with the evaluation framework indicate specific trends? If so, why? Maybe further investigation on this would provide some ideas for future exploration - which are also missing in the paper. ", "summary_of_the_paper": "The paper introduces a new approach for slice discovery by leveraging the advances in cross-modal embeddings. The paper also suggests a new evaluation framework to quantitively evaluate SDMs. ", "summary_of_the_review": "The paper introduces novel frameworks to evaluate and perform slice discovery. Though technical novelty in the individual parts are limited, the overall framework seem to be interesting and perform well according to the experimental results. The understanding of the paper suffers from the way how it is organised and should be further improved. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "W_ojUruamIr", "reviewer": "Reviewer_hQrb", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "General comments:\n\nThe paper addresses the problem of identifying on which subsets of data machine learning models make systematic errors. The authors term this as a \"slice discovery method\" (SDM). Two desirable properties for a SDM are outlined: First, providing a quantitative evaluation framework for measuring performance of SDMs and secondly, ensuring a solution that is \"coherent\". Here coherence is defined as being understandable by a domain experts, The proposed evaluation framework seems rather ad-hoc and poorly justified. The main idea, the DOMINO approach seems limited to images with captions. In the approach images and captions are separately embedded while preserving their similarity. This is followed by a mixture model which identifies errors in the model predictions, but it is not clear to me where the actual model predictions are trained. Also, the paper is often written in a confusing manner that makes it difficult to follow and understand the the contributions of the paper. Furthermore there is a lack of definitions for many of the terminology used in the paper.\n\nSpecific comments:\n\n- I find the term \"slice discovery method\" misleading, it is not commonly used term in this field.\n- The definition of the slice discovery problem (section 2) is rather imprecise and uses formulations such as \"exhibits degraded performance\". No precise definition is given what \"degraded\" means in this context. \n- After reading the definition of the slice discovery problem (section 2), I am still not clear whether a slice can refer to whole images only or part of the images (or frames in a video).\n", "summary_of_the_paper": "The paper propose a framework for identifying on which subsets of data machine learning models make systematic errors. The problem is cast in two parts: (1) identify a model that can be identify a subset of data and predict degraded performance of the machine learning model for this subset and (2) ensure that the identified subset is \"coherent\". The framework is evaluated on a number of classification tasks in computer vision and medicine.", "summary_of_the_review": "The paper addresses an interesting problem but is hampered by the ad-hoc nature of the approach and the lack of clarity in the problem formulation and writing.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "76xln7LyC1A", "reviewer": "Reviewer_si46", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n\nI believe that there is well backed motivation for work based off of the plentiful literature review.\nThere is novel integration of CLIP as well as SDMs previously not combined before\nA variety of datasets seem to show that the proposed method is useful.\nCode is available for reproducibility.\n\nCons:\n\nI would rather the the literature review section be in the intro rather than in the conclusion for flow. Then, the authors could summarize their work in the conclusion instead.\n\nAre textual descriptions of the Slices actually actionable for real life experts? I would like if physicians found the textual descriptions of the MIMIC and EEG dataset slices useful.\n", "summary_of_the_paper": "Recent studies have proposed automated slice discovery methods (SDMs), which leverage learned model representations to mine input data for slices, or important subgroups of data, on which a model performs poorly.\n\nAn ideal SDM should automatically identify: \n1. Slices that contain examples on which the model underperforms, or has a high error rate.\n2. Slices that contain examples that are coherent, or align closely with a human-understandable concept.\n\nThis is difficult because:\n1: No quantitative evaluation framework exists for measuring performance of SDMs; Existing SDM evaluations are either qualitative, performed on synthetic data, or consider only a small subset.\n2. Prior qualitative evaluations have demonstrated that existing SDMs often identify slices that are incoherent, even though they may satisfy the first ideal case.\n\nDomino:\nThe authors preset Domino, an SDM that leverages cross-modal embeddings and a novel error-aware mixture model to discover and describe coherent slices using natural language descriptions. The proposed method could also quantitatively compare SDMs, which has not been done before.\n\nStep 1. Embed: Encode inputs in a cross-modal embedding space with a function.\n\nStep 2. Slice:  identify underperforming regions in the cross-modal embedding space using an error-aware mixture model fit on the input embeddings, model predictions, true class labels using expectation maximization. I.e. input embeddings, class labels, and model predictions as independent based on slice. \n\nStep 3. Describe: Use the text embedding function ψtext learned in step (1) to generate a set of k natural language descriptions of the discovered slice.\n\nEvaluation Approach of 3 popular slide types:\nRare slice: To generate settings with rare slices, Construct a skewed dataset such that for a given class label Y, elements in subclass C occur with proportion α, where 0.01 < α < 0.1.\nCorrelation slice: Construct a dataset such that a linear correlation α exists between the target variable and other class labels, where 0.2 < α < 0.8.\nNoisy label slice. Construct dataset such that for each given class label Y, the elements in subclass C exhibit label noise with probability α, where 0.01 < α < 0.3.\n\nExperiments show that when cross-modal embeddings are provided as input, the error-aware mixture model often outperforms previously-designed SDMs.", "summary_of_the_review": "I would tend to accept this paper as it is novel enough and supported by empirical experiments.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Three experts reviewed this paper and all recommended acceptance. The reviewers liked that the work addressed a common problem in prior related work that it is hard to quantitatively evaluate slide discovery methods. Moreover, the proposed method achieves superior performance over prior arts. Based on the reviewers' feedback, the decision is to recommend the paper for acceptance. The reviewers did raise some valuable concerns, such as paper clarity, significance of the textual descriptions, that should be addressed in the final camera-ready version of the paper. The authors are encouraged to make the necessary changes to the best of their ability. We congratulate the authors on the acceptance of their paper!
Decision: Accept (Oral)