Venue: ICLR 2022 Conference

Reviews:

final_score: rating: null
review_content: I would like to thank the authors for addressing most of my concerns. I have adjusted my scores.
review_title: Updated scores

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: Recent studies have proposed automated slice discovery methods (SDMs), which leverage learned model representations to mine input data for slices, or important subgroups of data, on which a model performs poorly.

An ideal SDM should automatically identify: 
1. Slices that contain examples on which the model underperforms, or has a high error rate.
2. Slices that contain examples that are coherent, or align closely with a human-understandable concept.

This is difficult because:
1: No quantitative evaluation framework exists for measuring performance of SDMs; Existing SDM evaluations are either qualitative, performed on synthetic data, or consider only a small subset.
2. Prior qualitative evaluations have demonstrated that existing SDMs often identify slices that are incoherent, even though they may satisfy the first ideal case.

Domino:
The authors preset Domino, an SDM that leverages cross-modal embeddings and a novel error-aware mixture model to discover and describe coherent slices using natural language descriptions. The proposed method could also quantitatively compare SDMs, which has not been done before.

Step 1. Embed: Encode inputs in a cross-modal embedding space with a function.

Step 2. Slice:  identify underperforming regions in the cross-modal embedding space using an error-aware mixture model fit on the input embeddings, model predictions, true class labels using expectation maximization. I.e. input embeddings, class labels, and model predictions as independent based on slice. 

Step 3. Describe: Use the text embedding function ψtext learned in step (1) to generate a set of k natural language descriptions of the discovered slice.

Evaluation Approach of 3 popular slide types:
Rare slice: To generate settings with rare slices, Construct a skewed dataset such that for a given class label Y, elements in subclass C occur with proportion α, where 0.01 < α < 0.1.
Correlation slice: Construct a dataset such that a linear correlation α exists between the target variable and other class labels, where 0.2 < α < 0.8.
Noisy label slice. Construct dataset such that for each given class label Y, the elements in subclass C exhibit label noise with probability α, where 0.01 < α < 0.3.

Experiments show that when cross-modal embeddings are provided as input, the error-aware mixture model often outperforms previously-designed SDMs.
main_review: Pros:

I believe that there is well backed motivation for work based off of the plentiful literature review.
There is novel integration of CLIP as well as SDMs previously not combined before
A variety of datasets seem to show that the proposed method is useful.
Code is available for reproducibility.

Cons:

I would rather the the literature review section be in the intro rather than in the conclusion for flow. Then, the authors could summarize their work in the conclusion instead.

Are textual descriptions of the Slices actually actionable for real life experts? I would like if physicians found the textual descriptions of the MIMIC and EEG dataset slices useful.

summary_of_the_review: I would tend to accept this paper as it is novel enough and supported by empirical experiments.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper introduces a new approach for slice discovery by leveraging the advances in cross-modal embeddings. The paper also suggests a new evaluation framework to quantitively evaluate SDMs. 
main_review: The paper addresses interesting problems in the area of slice discovery, particularly underperforming clusters. Overall the Domino approach seems to work well for this task. There are ablation studies on the embeddings used and clustering algorithm chosen. The part about generating natural language explanation is not as convincing as it's ultimately using the text embeddings on a large corpus in a retrieval setting. Actually generating text from the given slice embeddings would have been more convincing. Currently the text seems to be restricted to single words from Fig 5. 
Though the idea introduced in the paper is quite interesting, the paper itself is organised in a very confusing manner. Related work is only introduced in section 6 on page 9 and seems incomplete. The Domino method is shown in Figure 1 on page 2 but the actual text describing it is on page 6. Another weakness is the novelty. Though the framework is novel, the individual models are not. Given the new evaluation framework, it would have been great to introduce some further technical novelty. 
In the experiments section, the results are described as-is without further interpretation. Do the results with the evaluation framework indicate specific trends? If so, why? Maybe further investigation on this would provide some ideas for future exploration - which are also missing in the paper. 
summary_of_the_review: The paper introduces novel frameworks to evaluate and perform slice discovery. Though technical novelty in the individual parts are limited, the overall framework seem to be interesting and perform well according to the experimental results. The understanding of the paper suffers from the way how it is organised and should be further improved.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper propose a framework for identifying on which subsets of data machine learning models make systematic errors. The problem is cast in two parts: (1) identify a model that can be identify a subset of data and predict degraded performance of the machine learning model for this subset and (2) ensure that the identified subset is "coherent". The framework is evaluated on a number of classification tasks in computer vision and medicine.
main_review: General comments:

The paper addresses the problem of identifying on which subsets of data machine learning models make systematic errors. The authors term this as a "slice discovery method" (SDM). Two desirable properties for a SDM are outlined: First, providing a quantitative evaluation framework for measuring performance of SDMs and secondly, ensuring a solution that is "coherent". Here coherence is defined as being understandable by a domain experts, The proposed evaluation framework seems rather ad-hoc and poorly justified. The main idea, the DOMINO approach seems limited to images with captions. In the approach images and captions are separately embedded while preserving their similarity. This is followed by a mixture model which identifies errors in the model predictions, but it is not clear to me where the actual model predictions are trained. Also, the paper is often written in a confusing manner that makes it difficult to follow and understand the the contributions of the paper. Furthermore there is a lack of definitions for many of the terminology used in the paper.

Specific comments:

- I find the term "slice discovery method" misleading, it is not commonly used term in this field.
- The definition of the slice discovery problem (section 2) is rather imprecise and uses formulations such as "exhibits degraded performance". No precise definition is given what "degraded" means in this context. 
- After reading the definition of the slice discovery problem (section 2), I am still not clear whether a slice can refer to whole images only or part of the images (or frames in a video).

summary_of_the_review: The paper addresses an interesting problem but is hampered by the ad-hoc nature of the approach and the lack of clarity in the problem formulation and writing.
review_title: null

Meta-review: Three experts reviewed this paper and all recommended acceptance. The reviewers liked that the work addressed a common problem in prior related work that it is hard to quantitatively evaluate slide discovery methods. Moreover, the proposed method achieves superior performance over prior arts. Based on the reviewers' feedback, the decision is to recommend the paper for acceptance. The reviewers did raise some valuable concerns, such as paper clarity, significance of the textual descriptions, that should be addressed in the final camera-ready version of the paper. The authors are encouraged to make the necessary changes to the best of their ability. We congratulate the authors on the acceptance of their paper!
Decision: Accept (Oral)