Concept-based Explanations for Out-of-Distribution Detectors
Authors: Jihye Choi
Jayaram Raghuram
Ryan Feng
Jiefeng Chen
Somesh Jha
Atul Prakash
Venue: Submitted to ICLR 2023 (2023)
Keywords: out-of-distribution detection
interpretability
concept-based explanations

Abstract: Out-of-distribution (OOD) detection plays a crucial role in ensuring the safe deployment of deep neural network (DNN) classifiers.
While a myriad of methods have focused on improving the performance of OOD detectors, a critical gap remains in interpreting their decisions.
We help bridge this gap by providing explanations for OOD detectors based on learned high-level concepts.
We first propose two new metrics for assessing the effectiveness of a particular set of concepts for explaining OOD detectors: 1) $\textit{detection completeness}$, which quantifies the sufficiency of concepts for explaining an OOD-detector's decisions, and 2) $\textit{concept separability}$, which captures the distributional separation between in-distribution and OOD data in the concept space.
Based on these metrics, we propose a framework for learning a set of concepts that satisfy the desired properties of detection completeness and concept separability, and demonstrate the framework's effectiveness in providing concept-based explanations for diverse OOD detection techniques.
We also show how to identify prominent concepts that contribute to the detection results via a modified Shapley value-based importance score.

Under review as a conference paper at ICLR 2023
CONCEPT-BASED EXPLANATIONS FOR OUT-OF-DISTRIBUTTO DETECTORS
Anonymous authors Paper under double-blind review
ABSTRACT
Out-of-distributioo (OOD) detection plays erucial role in ensuring the safe deploy- ment of deep neural network (DNN) classifiers. While a myriad of methods have focused on improving the performance OOD detectors, a critical gap remains interpreting their decisions. We help bridge this gap by providing explanations for OOD detectors based on learned high-level concepts. We first propose two new metrics for assessing the effectiveness of particular set of concepts for explaining OOD detectors: 1) detection completeness, which quantifies the sufficiency of concepts for explaining an OOD-detector's decisions, and 2) concept separability, which captures the distributional separation between in in-distribution and OOD data in the concept space. Based on these metries, we propose a framework for learning a set of concepts that satisfy the desired properties of detection completeness and concept separability, and demonstrate the framework 's effectiveness in providing concept-based explanations for diverse OOD detection techniques. We also show how to identify prominent concepts that contribute to the detection results via a modified Shapley value-based importance score.
INTRODUCTION
It is well known that machine learning (ML) models can yield uncertain and unreliable predictions on out-of-distribution (OOD) inputs, i.e., inputs from outside the training distribution (Amodei et al., 2016; Goodfellow al., 2015; Nguyen & O'Connor, 2015). The most common line of defense in this situation is to augment the MI model (e.g., a DNN classifier) with detector that can identify and flag such inputs as OOD. The MI model can then abstain from making predictions on such inputs (Hendrycks et al., 2019; Lin et al., 2021; Mohseni et al., 2020).
Currently, the problem of explaining the decisions of an OOD detector remains largely unexplored. Much of the focus in learning OOD detectors has been on improving their detection performance (Liu et al., 2020; Mohseni al., 2020; Lin et al., 2021: Chen et al., 2021: Sun et al., 2021; Cao & Zhang, 2022), but not improving their explainability. A potential approach would be to run existing interpretation method for DNN classifiers with ID and OOD data separately, and then inspect the difference between the generated explanations. However, it is not known if an explanation method that effective for explaining in-distribution class predictions will also be effective for OOD detectors. For instance, feature attributions, the most popular type of explanation (Sundararajan al., 2017; Ribeiro et al., 2016), may not capture visual differences in the generated explanations between ID and OOD inputs (Adebayo al., 2020). Moreover, their explanations based pixel-level activations may not provide the most intuitive form of explanations for humans.
This paper addresses the above research gap by proposing the first method (to our knowledge) to help human--unerstaann understandable way. We build upon recent advances in concept-based based explanations for DNN classifiers (Ghorbani et al., 2019; Zhou et al., 201 8a; Bouchacourt & Denoyer, 2019; Yeh et al., 2020), which offer an advantage of providing explanations in terms of high-level concepts for classification tasks. We make the first effort at extending their utility to the problem of OOD detection. Consider Figure which illustrates our concept-based explanations given inputs which are all classified as "Dolphin" by a DNN classifier, but detected as either ID or OOD by OOD detector. We observe that the OOD detector predicts certain input as ID when its concept-score patterns are similar to that of ID images from the Dolphin class. Likewise, the detector predicts input OOD when its concept-score patterns very different from that of
interpret the decisions of an OOD detector in a human-

---

Under review as a conference paper at ICLR 2023
1D (hod)
D (AwA)
1:30
1.00
Concegt score
0.23
(500) 2 1003
Concept score
0.75
0239 2-10
0.50
0.50
0.25
0.25
0:30
0.00
-0.25
-0.25
(a) Correct detection: ID (or OOD) dolphin image correctly detected as (or OOD).
Wrong detection: ID (or OOD) dolphin image falsely detected as OOD (or ID).
Our concept- explanation for the Energy OOD detector (Liu et al., 2020). On the x-axis, we present the top-5 important concepts that describe the detector's behavior given images classified as "Dolphin". Concepts C90 and C1 represent "dolphin-like skin" and "wwvy surface of the sea" respectively (see Figure 12b). ID profile shows the concept score patterns for ID images predicted Dolphin.
Figure I:
ID inputs. user can verify whether the OOD detector makes decisions based on the concepts that aligned with human intuition (e.g., C90 and CI), and that the incorrect detection (as in Figure Ib) is an understandable mistake, not a misbehavior of the OOD detector. Such explanations can help the user evaluate the reliability of the OOD detector and decide upon its adoption in practice.
We aim to provide general interpretability framework that is applicable to the wide range of OOD detectors in the world. Accordingly, research question we ask is: without relying on the internal mechanism of an OOD detector, can we determine a good set of concepts that are appropriate for understanding why the OOD detector predicts a certain input to be ID/OOD? A key contribution of this paper is to show that this can be done in an unsupervised manner without any human annotations.
We make the following contributions in this paper:
We motivate and propose new metrics to quantify the effectiveness of concept-basede explanations for a black-box OOD detector, namely detection completeness and concept separability ($ 2.2, $ 3.1, and 3.2).
We propose concept-learning objective with suitable regularization terms that, given an OOD detector for a DNN classifier, learns set of concepts with good detection completeness and concept separability ($ 3.3);
By treating given OOD detector as black-box, we show that our method can be applied to explain a variety of existing OOD detection methods. By identifying prominent concepts that contribute to an OOD detector's decisions via a modified Shapley value score based on the detection completeness, we demonstrate how the discovered concepts can be used to understand the OOD detector. ($ 4).
Related Work. In the literature of OOD detection, recent studies have designed various scoring functions based on the representation from the final or penultimate layers (Liang al., 2018: DeVries & Taylor, 2018), or combination of different internal layers of DNN model (Lee et al., 2018; Lin al., 2021; Raghuram al., 2021). A recent survey on generalized OOD detection can be found in Yang et al. (2021). Our work aims to provide post-hoc explanations applicable to wide range of black-box OOD detectors without modifying their internals. Among different interpretability approaches, concept- concept-based based explanation (Koh al., 2020; Alvarez Melis & Jaakkola, 2018) has gained popularity as is designed to be better-aligned with human reasoning (Armstrong et al., 1983; Tenenbaum, 1999) and intuition (Ghorbani al., 2019; Zhou al., 2018a; Bouchacourt & Denoyer, 2019; Yeh et al., 2020). There have been limited attempts to assess the use of concept-based explanations under data distribution changes such as adversarial manipulation (Kim et al., 2018) or spurious correlations (Adebayo et al., 2020). Designing concept-based explanations for OOD detection requires further exploration and is the focus of our work.
2 PROBLEM SETUP AND BACKGROUND
Notations. Let X' C Ro x denote the space of inputs x, where do is the number of channels and ag and by are the image size along each channel. Let y == {1,-.. L} denote the space of
We focus images, but the proposed method extends other domains.
2

---

Under review as a conference paper at ICLR 2023
output class labels Let Az denote the set of all probabilities over y (the simplex in L-dimensions). We assume that natural imputs to the DNN classifier are sampled from an unknown probability distribution Pin over the space X' x V The compact notation [n]] denotes {1. nj for positive integer n. Boldface symbols are used to denote both vectors and tensors. (x,x") denotes the standard inner-product between a pair of vectors. The indicator function 1 takes value (0) when the
condition e is true (false).
ID and OOD Datasets. Consider a labeled ID training dataset D {(x;, (Xx,y;); i- 1, N sampled from the distribution Pin- We assume the availability of an unlabeled training dataset D [x,, i 1.... _Nou from a different distribution, referred to as the auxiliary OOD dataset. Similarly, we define the ID test dataset (sampled from Pm)) as D and the OOD test dataset as D Note that the auxiliary OOD dataset D and the test OOD dataset D are from different distributions. All the OOD datasets are unlabeled since their label space is usually different from V.
OOD Detector. The goal of an OOD detector is determine if test input to the classifter is ID (i.e., from the distribution Pin); otherwise the input is declared to be OOD (Yang et al., 2021). Given a trained classifier f: X' + A,, the decision function of an OOD detector can be generally defined as (x, f)
(x, f) 2 7]. where S(x, f) R is the score function of the detector for input x and y is the threshold. We follow the convention that larger scores correspond to ID inputs, and the detector outputs of 1 and 0 correspond to ID and OOD respectively. We assume the availability pre-trained DNN classifier and a paired OOD detector that is trained to detect inputs for the classifier.
2.1 PROJECTION INTO CONCEPT SPACE
Consider pre-trained DNN classifier f AL that maps an input x to its corresponding predicted class probabilities. Without loss of generality, we can partition the DNN at con- volutional layer & into two parts, i.e., == h & where: 1) Φ X' Z := Rorbixd, is the first half of that maps an input x to the intermediate feature representation x(x). and 2) h Z AL is the second half of f that maps o(x) to the predicted class probabilities h(((x)). We denote the predicted probability of a class by (x) ( (T(())) and the prediction of the classifier by (x) argmax, (x).
Canonical World
Concept World
+
((x)
x(x)
h
h
Our work is based on the common implicit as- sumption of linear interpretability in the concept- based explanation literature, i.e.. high-level con- projected subspace of the feature representation space of the clas- sifier (Kim et al., 2018). 8). Consider projec- tion matrix = [c1, ... Cm] E mddem (with & de) that maps from the space Z into reduced-dimensee
Claseilication Completeness
f(x)
cepts lie in a linearly-proeected
Detection Completeness
8(x,F)
g
Figure 2: Our two-world view of the classifier and OOD detector.
concept space. consists of unit vectors, where (e.g.. "stripe" or "oval face"), ) and m the number of concepts. We define the concept score for x as the linear projection of the high-dimensional layer representation ((x) Racbexd, into the concept space (Yeh et al., 2020), i.e. (x(x) = Φ(x) C E RRobbemm. We also define a mapping from the projected concept space back to the feature space by a non-linear function g Roebexm + Raebexds
R. is referred to the concept vector representing the i-th concept
The reconstructed feature representation at layer é is then defined as @g.c(x) == g(ve(x)).
We flatten the first two dimensions of the feature representation, thus changing an x be x de tensor to an acb: de matrix, where af and by are the filter size and de is the number of channels.
3

---

Under review as a conference paper at ICLR 2023
2.2 CANONICAL WORLD AND CONCEPT WORLD
As shown in Fig. 2, we consider a "two-world" view of the classifier and OOD detector consisting of the canonical world and the concept world, which are defined as follows:
Canonical World. this case, both the classifier and OOD detector use the original layer representa- tion ((x) for their predictions. The prediction of the classifier is f(x) h( hn((())) and the decision function of the detector is (x,x, h a Φ) with a score function S(x, h o Φ).
Concept World. We use the following observation in constructing the concept-world formulation: both the classifter and the OOD detector can be modified to make predictions based on the recon- structed feature representation, i.e., using cg.)( instead of Φ(x). Accordingly, we define the corresponding classitier, detector, and score function in the concept world as follows:
(x( = h()),((x)) = h(g(vc(x)))
Da Xx f) = D,(x, h Øg.c) (x
Sen (x, = S(x, ho Øx.c) = S(x, hxx,hogom hogove).
(I)
We further elaborate on this two-world view and introduce the following two desirable properties.
Detection Completeness. Given a fixed algorithmic approach for learning the classifier and OOD detector, and with fixed internal parameters of f, we would ideally like the classifier prediction and the detection score be indistinguishable between the two worlds. In other words, for the concepts to sufficiently explain the OOD detector, we require Dem (x, f) to closely mimic D, f). Likewise, we require fx" to closely mimic f(x) since the detection mechanism of Dr,, is closely paired to the classifier. We refer to this property as the completeness of set of concepts with respect the OOD detector and its paired classifier. As discussed in $ 3.1, this extends the notion of classification completeness introduced by Ych al. (2020) an OOD detector and its paired classifier.
Concept Separability. To improve the interpretability of the resulting explanations for the OOD detector, we require another desirable property from the learned concepts: data detected as ID by Dr (henceforth referred to as detected-ID data) and data detected as OOD by D.,, (henceforth referred to as detected- OOD data) should be well-separated in the concept-score space. Since our goal is to help an analyst understand which concepts distinguish the detected-ID data from detected-OOD data, we would like to learn a set of concepts that have a well-separated concept score pattern for inputs from these two groups (e.g., the concepts "stripe" and "oval face" in Fig. ?? have distinct concept scores).
3 PROPOSED APPROACH
Given a trained DNN classifier f, a paired OOD detector DDy, and a set of concepts C, we address the following questions: 1) Are the concepts sufficient capture the prediction behavior of both the classifier and OOD detector? (see $ 3.1); 2) Do the concepts show clear distinctions in their scores between detected-ID data and detected-OOO data? (see 3.2). We first propose new metries for quantifying the set of learned concepts, followed by a general framework for learning concepts that possess these properties (see $ 3.3).
3.1 METRICS FOR DETECTION COMPLETENESS
Definition 1. Given trained DNN classifier f h o &b and set of concept vectors C, the classification completeness with respect to Pr (x, 3) is defined as (Yeh et al., 2020):
me (C) supg Ex.,) P. [y argmax,, by (
Or
E(x,y)- 1 1yy argmax, hy (x(x))]
ar
where = 1/L is the accuracy of a random L-class classifier.
The denominator of ny (C) is the accuracy of the original classifier f, while the numerator is the maximum accuracy that can be achieved in the concept world using the feature representation reconstructed from the concept scores. The maximization over the parameters of the neural network g that reconstructs the feature representation from the vector of concept scores.
4

---

Under review as a conference paper at ICLR 2023
Definition 2. Given trained DNN classifier f = h o Φ, trained OOD detector with score function S(x. f). and a set of concept vectors C, we define the detection completeness score with respect to the ID distribution (x. y) and OOD distribution Pau (x)( as follows:
Tr.s (C) supg AUC(h @g.c) - b,
(2)
AUC(h Φ) by
where AUC(F) is the area under the ROC curve of an OOD detector based on f, defined as AUC(T) (x,,))-7,, E P... f) > S(x', [[), and b, 0.5 is the AUROC of random detector.
The numerator term is the maximum achievable AUROC in the concept world via reconstructed features from concept scores. In practice, AUC(f) is estimated using the test datasets D and Des Both the classification completeness and detection completeness scores are designed to be in the range [0, 1]. However, this is not strictly guaranteed since the classifier or OOD detector in the concept world may empirically have a better (corresponding) metric on a given ID/OOD dataset. A completeness score close to 1 indicates that the set of concepts C are close complete in characterizing the behavior of the classifier and/or the OOD detector.
3.2 CONCEPT SEPARABILITY SCORE
Concept Scores. In Section 2.1, we introduced projection matrix C E Rddexm that maps Φ(x) to vc(x), and consists of m unit concept vectors C = [c1 ... Cm. .m. The inner product between the feature representation and concept vector is referred to as the concept score, and it quantifies how close input is to the given concept (Kim et al., 2018; Ghorbani et al., 2019). Specifically, the concept score corresponding to concept is defined as Ve. (x) := (ø(x), 3 = Φ(x) c, E gacbe The matrix of concept scores from all the concepts is simply the concatenation of the individual concept scores, i.e., (x(x) = Φ(x) C = [Ye, (x)...
...... Ve.. (x)] E xm We also define a rimension-reduced version of the concept scores that takes the maximum of the inner-product over each ag x be patch as follows: vx(x)" = [Te, (x), (x).... Te,., (x)] E R.", where Te, (x) = max;.9 (x)=.(()., c+)l E R. Here (x) is the feature representation corresponding to the (p. 9)-th patch of input (i.e., receptive field (Araujo et al., 2019)). This reduction operation is done to capture the most important correlations from each patch, and the m-dimensional concept score will be used to define our concept separability metric as follows.
We would like the of concept-score vectors from the detected-ID class Vin(C) = (x)(x), E DE U Drr class Vout (C) = (x). x E D U D Jaop JV (C), Your (C)) R define general measure of separability between the two subsets, such that a larger value corresponds to higher separability. We discuss a specific choice for Jap for which it is possible to tractably optimize concept separability part of the learning objective in Section 3.3.
(x,xf f) = 1}. and the set of concept-score vectors from the detected-OOD
fx 0} to be well separated. Let
Global Concept Separability. Class separability metrics have been well studied in the pattern recognition literature, particularly for the two-class case (Fukunaga, 1990b) 3, Motivated by Fisher's linear discriminant analysis (LDA), we explore the use of class-separability measures based on the within-class and between-clan scatter matrices (Murphy, 2012). The goal of LDA is to find projection vector (direction) such that data from the two classes are maximally separated and form compact clusters upon projection. Rather than finding an optimal projection direction, are more interested in ensuring that the concept-score vectors from the detected-II and detected-OOD data have high separability. Consider the within-class and between-class scatter matrices based (n) and Vourl (C), given by
S
(v- Min) (v
pin))
+
(v Houx) (v Mout) T CCV
Σ
(3)
VEVaIC)
S. (Moul Min) (poat Mmn)
where pm and Mou are the mean concept-score vectors from Vin and Von(C) respectively. We define the following separability metric based on the generalized eigenvalue equation solved by Fisher's LDA (Fukunaga, 1990b): Jup(()) == Juep (in (),, Vomt (C)) tr - S.]. Maximizing the above metric is equivalent to maximizing the sum of eigenvalues of the matrix S S,, which
In our problem, the two classes correspond detected-ID and detected-OOD.
5

---

Under review as a conference paper at ICLR 2023
in- turn ensures large between-class separability and small within-class separability for the detected- ID and detected-OOD concept scores. We refer to this as a global concept separability metric because it does not analyze the separability on a per-class level 4. The separability metric is closely related to the Bhattacharya distance, which is an upper bound on the Bayes error rate (see Appendix A.I).
3.3 PROPOSED CONCEPT LEARNING KEY IDEAS
Prior Approaches and Limitations. Among post-hoe concept-discovery methods for DNN clas- sifier with ID data, unlike Kim et al. and Ghorbani et al., that do not support imposing required conditions into the concept discovery, Yeh al. devised a learning-based approach where classifica- tion completeness and the saliency of concepts are optimized via regularized objective given by
argmax Cgg (x.y)~~P
log by (x(vc(x))) + Aexpt Respl
(5)
Here C and g (parameterized by neural network) are jointly optimized, and (es is regu- larization term used to ensure that the learned concept vectors have high spatial coherency and low redundancy among themselves (see (Yeh et al., 2020) for the definition).
While the objective (5) of Yeh et al. can learn set of sufficient concepts that have a high classification completeness score, we find that it does not necessarily replicate the per-instance prediction behavior of the classifier in the concept world. Specifically, there can be discrepancies in the reconstructed feature representation, whose effect propagates through the remaining part of the classifier. Since many widely-used OOD detectors rely on the feature representations and/or the classifier's ' predictions, this discrepancy in the existing concept learning approaches makes hard to closely replicate the OOD detector in the concept world (see Fig. 3). Furthermore, the scope of Yeh et al. is confined to concept learning for explaining the classifier's predictions based on ID data, and there is no guarantee that the learned concepts would be useful for explaining an OOD detector. To address these gaps, propose a general method for concept learning that complements prior work by imposing additional instance-level constraints on the concepts, and by considering both the OOD detector and OOD data.
Concept Learning Objective. We define a concept learning objective that aims to find a set of concepts C and mapping g that have the following properties: 1) high detection completeness w.r.t the OOD detector; 2) high classification completeness w.r.t the DNN classifier; and 3) high separability in the concept-score space between detected-ID data and detected-OOD data.
Inspired by recent works on transferring feature information from a teacher model to a student model (Hinton et al., 2015; Zhou et al., 2018b), we encourage accurate reconstruction of 2 based on the concept scores by adding a regularization term that is the squared Eg distance between the original and reconstructed representations Jnom (C, g) = Ex-P,, [1x(x) - (x) In order to close the gap between the scores of the OOD detector in the concept world and canonical world on a per- sarple level, we introduce the following mean- mean-square squared-error (MSE) based regularization:
Jmse (C, g) = E (S(x, h o Fg.c) )) -S(x, f))² +
E Xea
(S(x, h &g.c) - S(x, f))² (6)
x=PP
MSE terms are computed with both the ID and OOD data because we want to ensure that the ROC curve corresponding to both the score functions are close to each other (which requires OOD data). Finally, we include a regularization term to maximize the separability metric between the detected-II) and detected- -OD data in the concept-scone score space, resulting in our final concept learning objective:
argmax
E [log (y (x))] + deepl Rexpi( (C)
C.g (x.9))~ Pa
Amse Jmse Mass(C,, (C,g) Anorm J (C.g) + >sup Juep (C).
(7)
The À coefficients are non-negative Syper-parameters that are further discussed in Section 4. We note that both Jmse (C, g) and Jesp (C) depend on the OOD detector $ We use the SGD-based Adam optimizer (Kingma & Ba, 2014)) solve the learning objective. The expectations involved in the objective terms are calculated using sample estimates from the training ID and OOD datasets. Specifically. D and De are used to compute the expectations over P. and Pout, respectively. Our complete concept learning is summarized in Algorithm 1 (Appendix A.4).
*See Appendix A.2 and A.3 for per-class variations of detection completeness and concept separability. "This dependence may he obvious for the separability term, but it is clear from its definition.

---

Under review as a conference paper at ICLR 2023
4 EXPERIMENTS
We briefly describe the experimental setup here and provide additional details in Appendix B.
Datasets. For the ID dataset, we use Animals with Attributes (AwA) (Xian et al., 2018) with 50 animal classes, and split it into a train set (29841 images), validation set (3709 images), and test set (3772 images). We use the MSCOCO dataset (Lin et al., 2014) as the auxiliary OOD dataset D for training and validation. For the OOD test dataset D we follow a common setting in the literature of large-scale OOD detection (Huang & Li, 2021) and use three different image datasets: Places365 (Zhou al., 2017), SUN (Xiao et al., 2010), and Textures (Cimpoi et al., 2014).
Models. We apply our framework to interpret five prominent OOD detectors from the literature: MSP (Hendryeks & Gimpel, 2016), ODIN (Liang al., 2018), Generalized-ODIi (Hsu et al., 2020), Energy (Liu et al., 2020) and Mahalanobis (Lee et al., 2018). The OOD detectors are paired with the widely-used Inception-V3. model (Szegedy et al., 2016) (following the setup in prior works (Yeh et al., 2020; Ghorbani et al., 2019; Kim et al., 2018)) trained on the Animals- with-Attributes (AwA) dataset (Xian al., 2018). which has a test accuracy of 10%.
Metries. For each set of concepts learned with different OOD detectors and hyperparameters, we report the classification completeness M7 (n(C), detection completeness 17r,(( (C), and the relative concept separability metric (defined below). In contrast to the completeness scores that are almost always bounded to the range [0, 1]. it is hard to gauge the possible range of the separability score Juep (eppCC (C) (or Japp (C)) across different settings (datasets, classification models, and OOD detectors), and whether the value represents a significant improvement in separability. Hence, we define the relative concept separability, which captures the relative improvement in concept separability using concepts C compared to a different set of concepts C", as follows
Jsep C, C') = Median (L Jua (C) - JN (C')
(8)
[Epp(C")
y=
We choose C' to be the set of concepts learned by the baseline (Yeh et al., 2020). which is a special case of our learning objective when Ansse Anomm >sep = O. The set of concept C are obtained via our concept learning objective, with various combinations of hyperparameter values.
Test ODD dataset
000 desector
Hypar- parmmeeers
Cx(C)
Places
SUN
Textures
Ty. [C 1
J
(C
Jap.C, C
CT
0)
0.977 + 0.0006
0.774 + 0.0010
0.694
0.0153
0.782
D.0010
1,088 0.0175
0.593
0.0013
0.765 0.0157
MSP
010,0.1.0)
4: 0.0004
0.942 +: 0.0004
1.892
0.0093
0.946
3,074
0.920 0.0005
3.572 0.1292
(0,0,
0.980 + 0.0008
0.814 + 0.0008
2.533
0.0714
0816
0.0009
4.295 + 0,1048
0.773 0.0010
3.147 0.2076
(10, (.0.0.1.500
0.984 4: 0.0004
0.960 1: 0.0004
0.0854 2.756
0.0005 0.961
4.442 0.0830
0.937 0.0004
3,587 0.2145
00,0,0)
0.0000
0.742
0.01
0.745
0.710 0.0156
0.618
0.0013
0.501 00121
ODIN
[10". 00)
0.994 + 0.0004
0.951 + 0.0004
1.166
0.0303
0.958
0.0004
2,135 0.0450
0.934
0.0044
2.793 0.0865
(0,0,50)
0.987 1: 0.0004
0,899 1 0.0007
1.785
0.0669
0.911
0.0006
3.814 0.0768
0.793
0.0008
3,046 0.7855
(10%_0.1.50)
0.0005
0,973 0.0009
18133
0.0268
0.969
0.0010
4,000
0.0094
0.945
0.006
3.662 0 0.1005
0.988 1: 0.0004
0.769 1: 0.0004
0.506
0.0165
0,719
1: 0.0014
0,516 1: 0.0192
0.605
0.0013
0.558 O. 0,1683
General
[30%) 0.100
0.0004
0.951 + 0.0006
1.461
0.0321
0.960
0.0005
3.007
0.0316
0.940
0.0008
2.619 0.1077
ODIN
50)
0 1 0.0004
0.859 1. 0.0007
1$11
0.0685
0,803
0.0006
4.204
0,0159
0.526
0.0008
4,144 0.2216
[10") 501
1.990 1: 0.0005
0.971 1: 0.0010
0.0669 1,35
0.963: 0.0004
4.287
0.0284
0.951
0.0005
1 : 1921
(0,0,0)
0.977 + 0.0006
0.671 + 0.0012
0.453
0.0121
0.682
0.0012
0.675
0.0148
0.557
0.0014
0.521 0.0311
Energy
(1
0,993 + 0. 1005
0.965 +: 0.0004
1.266
0.0319
0.963 0.0004
2.125 0.0413
0.960 0.000
2.648 0.0896
(000,
0.982 + 0.0005
0.779 + 0.0010
1.920
0.0022
0.793 + 0.0009
3,659 + 0.0659
0.767
0.0010
4.397 0.2165
(3,0
0.0005
0.0003
0.0662
0.941 0.0005
3,421 0.0619
0.936 0.0005
3,917 1691
(0,0,0)
0.990 1 0.0007
0.715 1. 0.0011
0.571
0.0110
0,736
0.0011
0.822
0.0165
0.59]
0.000
0.564
0.0203
Mohala-
(0.1,0.1.0)
0.994 1 0.0004
0.950 1. 0.0009
1.532
0.0351
0.960
1. 0.0010
2.276
0.0466
0.938
0.0004
2.915 0.1132
nobis
(0,0,50)
0.985
0.0004
0,880
0.0005
2.550
0.883
0,0006
4,091
0.774 0.0007
4274 03305
10.1.0.1,50) 50)
0.992
1 0.0006
0.961
1 0.0005
2.616
0.0857
0.966 1. 0.0005
4.325 L. 0.0055
0.949 1. 0.0003
4,308 0.2011
Table 1: Results of concept learning with different parameter settings across various OOD detectors and test OOD datasets. The ID datasct is AwA for both training and test, and the auxiliary OOD dataset is MSCOCO. Hyperparameters are in the order of (Amses Anumm Awp)). and their values are set based on the scale of corresponding regularization terms in Eqn. (7), for specific choice of the OOD detector. Further investigation, including an ablation study on each regularization term can be found in Appendix B.2. Across the rows (for a given OOD detector and OOD dataset), the best value is boldfaced, and second best value is underscored. The 95% confidence intervals are estimated by bootstrapping the test set over 200 trials.
7

---

Under review as a conference paper at ICLR 2023
3rd Litbese koo
Inderbutnntaal
(a) Target distribution of S(x, f) in the canonical world.
(b) Reconstructed distribution of 9rem (x, f' in the concept world, us- ing concepts by (Yeh al., 2020).
(c) Reconstructed distribution of S (x, f) in the concept world, us- ing concepts by
Figure 3: Detection completeness and estimated density of OOD score S(x, f) from MSP detector. Concepts by ours are learned using Amse 10, Amom = 0.1, Auep = 50. Comparison made between AwA test set (ID; blue) vs. SUN (OOD; red).
4.1 EFFECTIVENESS OF OUR METHOD
In this subsection, we earry out experiments answer the following question: does our concept learning objective effectively encourage concepts to have the desired properties of detection com- pleteness and concept separability? Table summarizes the results concept learning with various combinations of hyperparameters for the proposed regularization terms in Eqn. (7): i) all the hy- perparameters are set to 0 (first row); ; ii) only the terms directly relevant to detection completeness (i.e., reconstruction error Jnorm C, g) and mean-squared error J (, g)) are included (second row); iii) only the term responsible for concept separability (Cep(C) is included (third row); iv) all the regularization terms are included (fourth row).
In all cases, we observe that the baseline achieves a high enough classification completeness score, but the lowest detection completeness and concept separability. This indicates that the concepts discovered by (Yeh et al., 2020) would be sufficient to describe the DNN classifier, but using the same set of concepts would be inappropriate for the OOD detector. In contrast, our method significantly improves the detection completeness (and even the classification completeness) by having Amse > O, Aporn > O and concept separability by having >sep > compared to the baseline. Moreover, the three terms make the best synergy together in almost all cases.
Detection completeness and accurate reconstruction of Z. Additionally, we observe whether the proposed evaluation metrics are well-aligned with the interpretability of the resulting concept- based explanations. In Fig. 3, we observe that concepts by (Yeh et al., 2020) with low detection completeness (1r. (C) = 0.782 for MSP and 7F.s Pt.s(C) (C) 0.682 for Energy) lead to strong mismatch between the score distributions on both ID data and OOD data. In contrast, concepts 0.961 for the MSP detector, and Tr.s (C)- 0.941 for the Energy detector) approximate the target score distributions more closely both ID data and OOD data. By reducing the performance gap of OOD detector between canonical world and concept world, it leads more accurate explanations for OOD detectors.
learned by method with high detection completeness (1r,s(()
4.2 CONCEPT-BASEE EXPLANATIONS FOR OOD DETECTORS
Contribution of each concept to detection. The proposed concept learning algorithm learns concepts for both the classifier and OOD detector considering all the classes, and we address the following question: how much does each concept contribute to the detection results for inputs predicted to a particular class?. Recent works have adopted the Shapley value from Coalitional Game theory literature (Shapley, 1953; Fujimoto et al., 2006) for scoring the importance of a feature subset towards the predictions model (Chen et al., 2018: Lundberg Lee, 2017: Sundararajan & Najmi, 2020). Extending this idea, we modify the characteristic function of the Shapley value using our per-class detection completeness metric (Eqn. (11) in Appendix A.2). The modified Shapley value a concept C, E C with respect to the predicted class [L] is defined
(m C'I 1)!
SHAP (T c.) ==
(Tk.s (C's (C'U (c:)) (C'),
(9)
m!!
O'soHe..
8

---

Under review as a conference paper at ICLR 2023
Yoh el al (043)
Yeh
3.00
1.00
10 haele
Conc score
6.75
00
Concept score
0.75
(Paace))
0.25
0.25
Yen et al (C29)
0.00
0:00
ID (AwA))
Oura (C-4
Yeh
unddla (00)
Conceept score
0.75
Concapt score
0.75
-1
0.50
Ours (C20)
ODD (Places)
0.22
0J5
1..00
0.00
C54
Figure 4: Concept-hased explanations for Energy OOD detector using concepts by Yeh al. (2020) vs. ours. Images are randomly selected from AwA test set (ID) and P Places (OOD), and all predicted class "Collie". ID profile shows the normal concept-score patterns for ID Collie images.
where C' is subset of C excluding concept C, and TÉ.s is the per-class detection completeness with respect to class j. This Shapley value captures the average marginal contribution of concept c; towards explaining the decisions of the OOD detector for inputs that are predicted into class j.
Eventually, we interpret the behavior of the given OOD detector by plotting the concept score patterns with respect to the concepts ranked by the above Shapley importance score. Figure illustrates the generated explanations given correctly-detccted inputs (ID / OOD input detected as ID/ OOD; first row figure), and incorrectly-deteete inputs OOD input detected as OOD HOD/ID. ID; second row of figure). Overall, we observe that the OOD detector predicts input ID when the concept scores show a similar pattern to the ID profile, or predicts an input as OOD when the concept score pattern is far from the ID profile. For instance, the fourth input is OOD image from P alae dataset but detected as ID, since its score for C54 (furry dog skin) is as high as usual ID Collie images (which is true in the image). Thus, we conclude this to be an understandable mistake by the OOD detector.
We also provide qualitative comparison between Yeh et al. and our method in the resulting expla- nations for OOD detector. We observe that Yeh et al. fails generate soually-distngguisabb explanations between detected-ID and detected-OOD inputs. The separation between the solid green bars and the orange bars in each figure becomes more visible in our explanations, which enables more intuitive interpretation for human users, and this reflects our design goal for concept separability. is also noteworthy that our concepts that are most important to distinguish IID Collie from OOD Collie (i.e., C54 and C30) are more specific, and finer-grained characteristies of Collie, while Yeh et al. finds concepts that are vaguely similar to the features of dog, but rather generic (i.e., C43 and C29). This is the reason require more number of concepts to achieve detection completeness and concept separability, compared to solely considering the classification completeness 6
CONCLUSION
In this work, we make a first attempt developing unsupervised and human-ine inmaan-ntterrretbble explana- tion method for black-box OOD detectors based on high-level concepts derived from the internal layer representations of a (paired) DNN classifier. We propose novel metrics viz. detection completeness and concept separability to evaluate the completeness (sufficiency) and quality of the learned concepts for OOD detection. The proposed concept learning method is quite general and applies broad class of off-the-shelf OOD detectors. Through extensive experiments and qualitative examples, we demonstrate the practical utility of our method for understanding and debugging OOD detector. We discuss additional aspects of our method such as the choice of auxiliary OOD dataset, human subject study, and societal impact in Appendix E.
'In Figure 4, after concept learning with m 100 and duplicate removal, we find 44 non-redundant concepts for Yeh al. (Aese A Auep 0), and 100 distinct concepts for ours (Ause Aumm 0.1, Aeep 10).

---

examples in Under review as a conference paper at ICLR 2023
REFERENCES
Julius Adebayo, Michael Muelly, Ilaria Liccardi, and Been Kim. Debugging tests for model explana- tions. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Informa- tion Processing Systems, 2020, URL ht http///proccee...../e.//o neurinn ec/ ct/paper/2020, hash//
html.
David Alvarez Melis and Tommi Jaakkola. Towards robust interpretability with self-explaining neural networks. In S. Bengio, H. Wallach, H. Lamchelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.). Advances in Neural Information Processing Systems, volume 31. Cur- ran Associates, Inc., 2018. URL 2at www.diinnnn.........oo..muu.. www.ins. /ttpr/ps.cc//aaee///////... (0O00e92288900446222333333389909
Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete problems safety. arXiv preprint arXiv: 160,006655,, 2016.
Andre Araujo, Wade Norris, and Jack Sim. Computing receptive fields of convolutional neural networks. Distill, 2019. doi: 10.2391 5/isstiliill.. https://isstill putp://ssiiilllp//00/000 wttpo/////ittiingeeeeoeooo receptive-fielde
Sharon Lee Armstrong, Lila R Gleitman, and Henry Gleitman. What some concepts might not be. Cognition, 13 (3):263-308, 1983.
Anil Bhattacharyya. On measure of divergence between two statistical populations defined by their probability distributions. Bull. Calculta Math. Soc., 35:99-100 109, 1943.
Diane Bouchacourt and Ludovic Denoyer. Educe: Explaining model decisions through unsupervised concepts extraction. arXiv preprint 1905, /852, 2019.
Senqi Cao and Zhongfei Zhang. Deep hybrid models for out-of-distribution detection. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pp. 4733-4743, 2022.
Jianbo Chen, Le Song, Martin J Wainwright, and Michael Jordan. L--shalley and c-shapley: Efficient model interpretation for structured data. In International Conference on Learning Representations, 2018.
Jiefeng Chen, Yixuan Li, Xi Wu, Yingyu Liang, and Somesh Atom: Robustifying out-of distribution detection using outlier mining. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 430-40400 445. Springer, 2021.
Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3606-3613, 2014.
Terrance DeVries and Graham W Taylor. Learning confidence for out-of-distribution detection in neural networks. arXiv preprint 1rXiv: 18011888628533 2018.
Katsushige Fujimoto, Ivan Kojadinovic, and Jean-Luu Marichal. Axiomatic characterizations of probabilistic and cardinal- probabilltto interaction indices. Games and Economic Behavior, 55(1):72-99, 2006.
Keinosuke Fukunaga. Introduction to Statistical Pattern Recognition, chapter 3, pp. 97-103. Academic Press,
edition, 1990a.
Keinosuke Fukunaga. Introduction Statisticul Puttern Recognition, chapter 10, pp. 446-451. Academic Press, edition, 1990b.
Amirata Ghorbani, James Wexler, James Zou, and Been Kim. Towards automatic concept- based explanations. Advances Neural Information Processing Systems, 32:9277-9286. 2019.
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In International Conference on Learning Representations, 2015.
Dan Hendryeks and Kevin Gimpel. A baseline for detecting misclassified and neural networks. arXiv preprint arXiv:1610.02136 2016.
Dan Hendrycks, Mantas Mazeika, and Thomas G. Dictterich. Deep anomaly detection with outlier exposure. In
7th International Conference on Learning Representations (ICLR). OpenReview.net, 2019, URL ht //pppeeeviee.... net htw.ee///oooummiccccCaaeaeeeeee
Dan Hendrycks, Andy Zou. Mantas Mazeika, Leonard Tang, Bo Li, Dawn Song, and Jacob Steinhardt. Pixmix: Dreamlike pictures comprehensively improve safety measures. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pp. 16783-16792, 2022.
10

---

Under review as a conference paper at ICLR 2023
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint arXiv: 02531, 2015.
Yen-Chang Hsu, Yilin Shen, Hongxia Jin, and 7solt Kira. Generalized odin: Detecting out-of-distributin image without learning from out-of-distributio data. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recegnition, pp. 10951-10960, 2020.
Rui Huang and Yixuan Li. MOS: Towards scaling out-of-distribution detection for large semantic space. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp- 8710-8719, June 2021.
Been Kim, Martin Wattenberg, Justin Gilmer, Carric Cai, James Wexler, Feraanda Viegas, et al. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (TCAV). In International conference on machine learning. pp. 2668 2677. PMLR, 2018.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6680. 2014.
Pang Wei Koh, Thao Nguyen, Yew Siang Tang. Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. Concept bottlencek models. International Conference on Machine Learning, pp. 5338-5348. PMLR, 2020.
Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting out-of- distribution samples and adversarial attacks. Advances in neural information processing systems, 31, 2018.
Shiyu Liang, Yixuan Li, and R Srikant. Enhancing the reliability of out-of-distributinn image detection in neural networks. In International Conference Learning Representations, 2018
Tsung-Y Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsofi COCO: Common objects in context. In European Conference on Computer Vision ECCV, volume 8693 of Lecture Notes in Computer Science, pp. 740-755. Springer, 2014. doi: 10. 1007/978-3-33 10602- I' L8.. URL ht https: //oi. . org 1007/ 978-- 3- 3-1--199--11 109---00---111 48.
Ziqian Lin, Sreya Dutta Roy. and Yixuan Li. MOOD: Multi-level out-of-distribution detection. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pp. 15313-15323, 2021.
Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li. Energy-based out out-o--distribution detection. Advances in Neural Information Processing Systems, 33, 2020.
Scott M Lundberg and Su-In Lee. A unified approach to interpreting model predictions. In Proceedings of the 3t international conference on neural information processing systems, pp. 4768-478888 4777, 2017.
Sina Mohseni, Mandar Pitale, JBS Yadawa, and Zhangyang Wang. Self-supervised learning for generalizable out-of-distribution detection. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 5216-5223, 2020.
Kevin P Murphy. Machine Learning: A Probabilistic Perspective, chapter 8, pp. 271-274. MIT press, 2012.
Khanh Xuan Nguyen and Brendan O' Connor. Posterior calibration and exploratory analysis for natural language processing models. In EMNLP, 2015.
Jayaram Raghuram, Varun Chandrasckaran, Somesh Jha, and Suman Banerjee. A general framework for detecting anomalous inputs DNN classifiers. In International Conference on Machine Learning, pp- 8764-8775. PMLR, 2021.
Marco Tulio Ribeiro. Sameer Singh, and Carlos Guestrin. "why should i trust you?" explaining the predictions of any classifier. Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pp. 1135-1144, 2016.
LS Shapley. value for n-person games. Contributions the Theory Games, (28):307-317, 1953.
Yiyou Sun, Chuan Guo, and Yixuan Li. ReAct: Out-of-distribution detection with recti- fied activations. 144 157, 2021.
In Advances in Neural Information Processing Systems (NeurIPS), pp. wtw.ooooeiig...nurrii........... wwwrrps........ peeurips.ce/papeeee wttp:ccppppp........///// /hash/ 3117779335333..
URL -ab.aaaataaatt..bbbe....aaaaaa.a .html.
21844d604843322cccdea.
Mukund Sundararajan and Amir Najmi. The many shapley values for model explanation. In International Conference on Muchine Learning, pp. 9269-9278. PMLR, 2020.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks, In International Conference on Machine Learning, pp. 3319-3328. PMI.R, 2017.
11

---

Under review as a conference paper at ICLR 2023
Christian Szegedy. Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 2818-2826, 2016.
Joshua Brett Tenenbaum. Bayesian frameworkk for concept learning. PhD thesis, Massachusetts Institute Technology, 1999.
Yongqin Xian, Christoph Lampert, Bernt Schiele, Zeynep Akata. Zero-shor learning comprehensive evaluation of the good, the bad and the ugly. IEEE transactions on pattern analysis and machine intelligence, 41(9):2251-22) -26622 2018.
Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale seene recognition from abbey to zoo. In 2010 IEEE computer sociery conference on computer vision and pattern recognition, pp. 3485-344 3492. IEEE, 2010.
Jingkang Yang, Kaiyang Zhou, Yixuan Li, and Ziwei Liu. Generalized out-of-distributiin detection: A survey, CoRR, abs/2110. 11334, 2021. URL https: :tap::://aa/ //tp//vaavv//////a////// wtgg/bbs////////./ooo/. 10..1.1111 1 11344.
Chih-Kuan Ych, Been Kim, Sercan Arik, Chun-Liang Li, Tomas Pfister, and Pradeep Ravikumar, On completeness-avve aware concept-based explanations in deep neural networks. Advances in Neural Informa- tion Processing Systems, 33:20554-20565. 2020.
Bolci Zhou, Agata Lapedriza, Aditya Khosla, Audc Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE transactions on pattern analysis and machine intelligence, 40(6); 1452-1464, 2017.
Bolei Zhou, Yiyou Sun, David Bau, and Antonio Torralha. Interpretable hasis decomposition for visual explanation. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 119-134, 2018a.
Guorui Zhou, Ying Fan, Runpeng Cui, Weijie Bian, Xiaoqiang Zhu, and Kun Gai. Rocket launching: universal and efficient framework for training well well-performing light net, In Thirty-second AAAI conference on artificial intelligence, 8b.
12

---

Under review as a conference paper at ICLR 2023
Appendix
In Section A, we discuss the connection of the proposed concept separability Bhattacharya Distance, and the per-class variations of detection completeness and concept separability, followed by the overall algorithm for concept learning. In Section B, we provide the detailed setup for the experiments, In Section C, we discuss whether our concept learning objective remains effective even when synthesized auxiliary OOD dataset similar to target ID data used. In Section D, we illustrate additional examples of our concept- concept-based based explanations.
A CONCEPT LEARNING
A.1 CONNECTION TO THE BHATTACHARYA DISTANCE
We note that the proposed separability metric in Section 3.2 is closely related to the Bhattacharya distance Bhattacharyya (1943) for the special case when the concept scores from both ID and OOD data follow a multivariate Gaussian density. The Bhattacharya distance is well known measure of divergence between two probability distributions, and it has the nice property of being an upper bound the Bayes error rate in the two-class case Fukunaga (1990a). For the special case when the concept scores from both ID and OOD data follow a multivariate Gaussian with a shared covariance matrix, can be shown that the Bhattacharya distance reduces to the separability metric:
Jepp C) == Jsep ( (Vi (C), Your (C)) = tr tr(s S.].
(10)
(ignoring scale factors).
A.2 PER-CLASS DETECTION COMPLETENESS
Per-Class Variations. We propose per-class measures for the detection completeness, which are obtained by simply modifying Tr.s((r ) in Eqn. (2) and (Cp based on the subset of and OOD data whose predictions are class y E [L]. We refer to these per-class variations as per-class detection completeness (denoted by ni.s (C)).
Definition 3. Given trained DNN classifier f = h Φ. a trained OOD detector with score function S(x, f), and a set of concept vectors C, the detection completeness relative to class [Z]] with respect to the ID distribution Pin Pin(X,0 y) and OOD distribution Pom (x) is defined as
m (C) = sup& AUC (h ex.c) by
(11)
AUC o Φ) b,
where AUC' (ho@g. is is the AUROC of the detector conditioned on the event that the class predicted by the concept-worid classifier h Φg.C is (note that the denominator has the global AUROC). The baseline AUROC b, is equal to 0.5 as before. This per-class detection completeness is used in the modified Shapley value defined in section 4.2.
A.3 PER-CLASS CONCEPT SEPARABILITTY
In section 3.2, we focused on the separability between the concept scores of ID and OOD data without considering the class prediction of the classifier. However, it would be more appropriate to impose a high separability between the concept scores on a per-class level. other words, we would like the concept scores of detected-ID and detected-OOD data, that are predicted by the classifier into any given class y [L] to be well separated. Consider the set of concept-score vectors from the detected-ID (or detected-OOD) dataset that are also predicted into class y;
V (C) = (x)(x), x E D Drr D2(x, = and (xx) = y} Vou (C) = (x)(x), x E D D D,(x, = and (x) = y}.
(12)
We extend the definition of the global separability metric in Eq. (10) to given predicted class y E [T.] as follows
scp (C) = Juep (C), V. (C)) = (r(())) sg]
( /1/) (S%) ( m²))
(13)
13

---

Under review as a conference paper at ICLR 2023
We refer to these per-class variations per-class concept separability. The scatter matrices 5% and Sy are defined similar to Eq. (3), using the per-class subset of concept scores V (C) or Vou(( and the mean concept-scone vectors from the detected-ID and detected-OOD dataset are also defined at per-class level.
A.4 ALGORITHM FOR CONCEPT LEARNING
To provide the readers with clear overview of the proposed concept learning approach, we include Algorithm 1. Note that in line 7 of Algorithm 1, the dimension reduction step in fx. }} and (Cm(CC (Vc(x), x E D out (x. 0} involves the maximum function, which is not differentiable; specifically, the step Uo (x) maxp.9 (() c;)l. For calculating the gradients (backward pass), we use the log sum exp function with temperature parameter to get differentiable approximation of the
(C) = [Vo(x)) x): E D U D
maximum function, i.e., maxp.q (x), c.)l s a log Ep., exp ( 1(p*. c.)l)
as - 0.
In our experiments, we set the temperature constant a 0.001 upon checking that the approximate value of Fe, (x) is sufficiently close to the original value using the maximum function.
Algorithm 1 Learning concepts for OOD detector
Dull) classifier INPUT: Entire training set D {D in? Dumt entire validation set D {Dm yy
in
detector D.y.
INITIALIZE: Concept vectors = [c] ......c. cm] and parameters of the network g.
OUTPUT: C and g-
1: Calculate threshold for D.y using D the score at which true positive rate is 95%.
2: for t = 1..I epochs do
3: Compute the prediction accuracy of the concept-world classifier feom using D
4: Compute the explainability regularization term as defined in Yeh et al. (2020).
5: Compute difference of feature representation between canonical world and concept world (i.e.
Joomn C, g)).
6: Compute difference of detector outputs between canonical world and concept world using Eqn. (6).
7: Compute (C) and (C) using D D.y and C.
8: Compute separability between Vnn (C) and Vout C)) using Eqn. (10) Eqn. (13).
9: Perform a batch-SGD update of C and using Equ. (7) as the objective.
A.5 ACCURATE RECONSTRUCTION OF CLASSIFIER OUTPUTS
We have performed additional experiments to understand if the proposed method can provide im- provements in the classification setting. Let C₁ denote the concept matrix learned by the method of Yeh et al. Let C2 denote the concept matrix learned by our method with Amse Xeep = 0 and Anorm = 0.1 (set based on the scale of the regularization term Jnorm). The idea is that we exclude the terms in the concept- meaeppttlernnng objective (Eqn. 7) that depend the OOD detector, but include the f2 norm based reconstruction error of the layer representation. To evaluate the utility of these two sets of concepts for classification, calculated the per-sample Hellinger distance between the predicted class probabilities of the original classifier and the concept-world classifier (based on either C or C2). Figure 5 compares the empirical distribution of the Hellinger distance for both sets of concepts C and C2. We observe that the distribution is more skewed towards zero with higher density near zero and a shorter (right) tail in the case of C2 (red curve) compared to C₁ (blue curve). This suggests that the class predictions are more accurately reconstructed by the concepts learned using our method with only the reconstruction error-based regularization. This can in-turn benefit the concept-based explanations for the classifier.
A.6 ACCURATE RECONSTRUCTION OF OOD SCORES
In addition to Figure where we compared the reconstruction accuracy of OOD scores using concepts by (Yeh al., 2020) and ours, Figure 6 confirms that the same observation applies to Energy detector.
14

---

Under review as a conference paper at ICLR 2023
Yel et nl.
Avams Aeue O.A
Density
0
1.0
Hellinger distannce
Figure Examples for correct detection
CAEEAA FHO
1.0
Dosger
8:10
(a) Target distribution of S(x, f) in the canonical world.
(b) Distribution of 5500 (x, f) in the concept world, using concepts
(c) Distribution of 5 (x, f) in the concept world, using concepts by
by (Yeh al., 2020).
Figure 6: Detection completeness and estimated density of OOD score S(x, f) from Energy detector. Comparison is made between AwA test (ID; blue) vs. SUN (OOD; red).
A.7 CONCEPT SEPARABILITY AND VISUAL DISTINCTION IN EXPLANATIONS
In Fig. 7, we take the average of concept scores (C) (or Vour (C) (Cn) among the inputs that are predicted as class y. and detected as (or OOD) by Energy detector as an example. We can observe noticeably distinguishable pattern between detected-ID and detected-OOD concept scores when using concepts with higher concept separability (Jup C, C') = 3.421), ), compared to those of low concept separability (Jaep(C C, C') 0.675) by Yeh et al. (2020). These observations confirm motivation for the concept separability metric - that a higher value of the concept separability metric enables better visual distinction between the concept score patterns, suggesting better interpretability for humans.
design
B IMPLEMENTATION DETAILS
We all experiments with Tensorflow, Keras and NVDIA GeForee RTX 2080Ti GPUs. We use test set bootstrapping with 200 runs to obtain the confidence interval for each hyperparameter set of concept learning.
B. EXPERIMENTAL SETTING.
OOD Datasets. For the auxiliary OOD dataset for concept learning (Dit). we use the unlabeled images from MSCOCO dataset (120K images in total) Lin et al. (2014). We carefully curate the dataset to make sure that no images contain overlapping animal objects with our ID dataset (i.e., 50 animal classes of Animals-withe with-Attributes Xian et al. (2018)), then randomly sample 30K images. For OOD datasets for evaluation (Die), we use the high-resolution image datasets processed by Huang and Li Huang & Li (2021).
15

---

Under review as a conference paper at ICLR 2023
Detected-ID
Detected-ID
Concapt
04
Concept
04
C41
C17
C36
C44
C30
CH8
Detected-OOD
Detected-OOD
0.8
08
0.6
04
-02
C35 OAL
(a) Energy, baseline (0.675)
(b) Energy, ours (3.421)
Figure 7: Concept separability and visual distinction in the concept score patterns. For the class "Giraffe", we compare the concept score patterss using two different sets of concepts. Left:
Averaged scores of top- 10 important concepts out the concepts learned by Ych et al. (2020)). Right: Averaged scores top- 10 important concepts out of the concepts learned by our method ( Amee = 1, Anamu = 0.1, Aep 50 with Energy detector). Concept importance is measured using the Shapley value of Eqn. (9).
Hyperparameters for Concept Learning. Throughout the experiments, we fix the number of concepts to 100 (unless specifically mentioned otherwise), and following the implementation of Yeh et al. (2020), we set Aexp = 10 and to be two-layer fully-com colly-comnected neural network with 500 neurons in the hidden layer. We learn concepts based on feature representations from the layer right before the global max-pooling layer of the Inception-V V3 model. After concept learning with m concepts, we remove any duplicate (redundant) concept vectors by removing those with dot product larger than 0.95 with the remaining concept vectors Ych al. (2020).
B.2 ADDITIONAL RESULTS
Ablation study for concept learning. We perform an ablation study that isolates the effect of each regularization term in concept learning objective (Eqn. 7) towards evaluation metrics: classification completeness, detection completeness, and relative concept separability. We also observe the coherency among the learned concepts by varying Amsc and A Coherency of concepts was introduced by Ghorbani et al. Ghorbani et al. (2019) to ensure that the generated concept-based explanations are understandable humans. It captures the idea that the examples for a concept should be similar to each other, while being different from the examples corresponding to other concepts. For the specific case of the image domain, the receptive fields most correlated to a concept (e.g., "stripe pattern") should look different from the receptive fields for different concept, (e.g., "wavy surface of sea"). Yeh et al. Yeh al. (2020) proposed quantify the coherency of concepts as
1
(
(14)
mK
where Te, is the set of K-nearest neighbor patches of the concept vector c; from the training set Drr
We use this metric to quantify how understandable our concepts are for different hyperparameter choices. Figure 8 shows that aligned with our intuition, large Amse helps to improve the detection completeness. Having non-zero Amse also helpful to improve the classification completeness even further, and surprisingly concept separability On the other hand, on the right side of Figure 8, we observe that large relative concept separability with large Avep comes at the expense of lower detection completeness and coherency. Recall that
well, without sacrificing the coherency of concepts.
16

---

Under review as a conference paper at ICLR 2023
1.2
0.8
0.6
Clasefiction Comprsteness
Relethve Concept Sepwraiiity
Detection on Completenens
Conorency
0.4
Casaiical Completenna
Concept Separasiliy
Debertion Complete Coherency
0.0
40
62
Asea
(a) Ablation study varying Amse; we set Anorm
(h) Ablation study varying Axepi set Amse
0.1, Amp =
0, Amorm
Figure 8: Ablation study with respect to Juse (C, g) and (C). We fix 100, Aexpl 10. and the OOD detector used for concept learing and evaluation is Energy Liu et al. (2020)
when visualizing what each concept represents for human's convenience, we apply threshold 0.8 to only presents (see Figure 10). Low coherency with respect to Eqn. 14 (i.e., 0.768 with Xsep = 75) means that there are much less number of examples that can pass the threshold, meaning that users can hardly understand what the concepts at hand entails. This observation suggests that one needs to balance between concept coherency and concept separability depending on which property would be more useful for specific application of concepts.
Transferability of concepts across OOD detectors. Our work essentially suggests to use different set of concepts for specific target OOD detector, Jmse (C, g) and Jspp in Eqn. (7) depend choice of OOD detector. In practice, however, one might not have enough computational capacity to prepare multiple sets of concepts for all type of OOD detectors at hand. Here, we inspect whether the concepts targeted for certain type of OOD detector are also good to be used for other OOD detectors.
We explore the transfcrability of concepts targeted to MSP Hendrycks & Gimpel (2016) detector in Table 2a, and Energy Liu al. (2020) Table 2b. Not surprisingly, we observe that concepts targeted for Energy yields the best detection completeness score when tested with the same type of OOD detector, but still make meaningful improvement with other detectors as well. When comes to relative concept separability, it is transferred even better across different OOD detectors. For instance, the concepts lead to Japp (C, C') = 0.862 with Tres, the best relative concept separability is achieved with ODIN detector (i.e., Jspp C, C') 0.862) and which is even higher than the best results we could obtain using the set of concepts targeted for ODIN (i.e., Jsep (C, C') = 0.414 with Amse
Anamn
Avep 50 in Table 1).
D
ODD desector
OOD
Metrics
MSP
ODIN
Energy
Mahal
MSP
ODIN
Energy
Mahal
(C)((C)
0.959
0.952
0.938
0.947
(Cr,(())
0.956
0.954
0.971
0.954
P1 Places
Ca (*)
0.327
0.288
0,361
0.338
Places
Ca,
0.417
0.455
0.365
0.410
(C)
0.961
0.954
0.945
0.953
(r,((C)
0.949
0.948
0.970
0.950
Jan(C, C*)
0.266
0.294
0.390
0.351
0.355
0.286
0,400
0.353
(C)(C)
0.938
0.946
0.932
0.930
4,,(C)
D.943
0.964
0.947
Textures
C²)
Textuzes
C=)
0.494
(C)
0.946
0.946
0.933
0.930
(C)
0.943
0.939
0.973
0.940
(Naturaliat
Naturalist
Sop(C,
0.286
0.181
0.229
0.197
C,
0.283
0.448
0.280
0.326
(a) Concepts targeted for MSP with Amse
(b) Concepts targeted for Energy with Amse
10, Aporm
0.1, Aup = 50
1, Anum = 0.1, Awap
50
Table 2: Transferability of concepts across different OOD detectors.
17

---

Under review as a conference paper at ICLR 2023
C DISCUSSION ON THE CHOICE OF AUXILIARY OOD DATASET IN CONCEPT LEARNING
Under circumstances where having access auxiliary OOD dataset for concept learning is not feasible, we suggest that one could use generative methods to generate synthetic dataset, or apply data augmentation techniques (Hendrycks et al., 2022). Figure shows an example of AwA image augmented by Hendryeks et al. (2022).
Figure 9: Random example of augmented AwA dataset. Left: original image in AwA train set. Right: corresponding image augmented by Hendrycks et (2022).
We evaluate the effectiveness of our concept learning objective when such augmented AwA train is used as auxiliary OOD dataset. Table C illustrates that the generated concepts with augmented AwA (i.e., OOD data close to target ID data) have comparable detection completeness and concept separability compared to when MSCOCO (i.e., OOD data far from ID data) was used. But still, further evaluation on generated concept-based explanations with different choice of auxiliary OOD dataset remains an interesting research question.
00D desector
Hyper- parametors
[C1
Places
SUN
Textures
CCT
(r,((0)
Jay CCT
Energy
(1,01150))
6955
0.0066
0.940
00055
1.746
0,0722
0.910
0.80555
3,0703 + 0.580
0,927 0.005
3.477 0,1419
Table 3: Results of concept learning with augmented AwA train set as auxiliary OOD in concept learning.
D EXPLANATIONS
D.1 IMPORTANT CONCEPTS FOR EACH OOD DETECTOR
We show additional examples for the top-ranked concepts by SHAP c;) in Figure ??. For each figure with fixed choice of class prediction, we present receptive fields from ID test set corresponding to top concepts that contribute the most to the decisions of each OOD detector. All receptive fields passed the threshold test that the inner product between the feature representation and the corresponding concept vector is over 0.85.
Moreover, in Fig. 11, compare the important concepts discovered by the baseline method Yeh et al. (2020) (denoted "baseline") vs. ours. With the baseline, when the learned concepts are solely intended for reconstructing the behavior of the classifier, we observe that interpretation of both the classifier and OOD detector depends on a common set of concepts (i.e., concepts 32, 10, and 47). On the other hand, the concepts learned by our method focus on reconstructing the behavior of both the OOD detector and the classifier. In this case, we observe that a distinct set of important concepts are selected for classification and OOD detection. We also observe that our method requires more concepts in order to address the decisions of both the classifier and OOD detector. For instance, the number of concepts obtained by our method and the baseline are 78 and 53 (respectively), out of a total 100 concepts after the duplicate removal of concept vectors. In short, when the concepts are only targeted at explaining the DNN classifier (as in the baseline Yeh et al. (2020)), the behavior of the OOD detector is merely described by the common set of concepts that important for the DNN classifier. On the other hand, when not only the DNN classifier but also the OOD detector is taken into consideration during concept learning (i.e., method), obtain more diverse and
18

---

Under review as a conference paper at ICLR 2023
Detected-ID ID Data
Detected-ID Data
Concept 28: 0.385
Concept 42: 0.241
AwA
AwA
Concept 0.087
Concept
SUN
SUN
Concept 0,048
Concept 0.023
Detected-OOn -OD Data
Detected-OOG -Oo Data
Concept 70: 0.020
Concept 21: 0.022
AwA
SUN
Concept 54: 0.014
Concept
SUN
Figure 10: Top-6 important concepts for Energy with respect. Left: class "Sheep". Right: class "Giraffe"
Baseline
Ours
Classified to "Buffalo"
Classification Concept 32: 0.850
Detection Concept 32: 0.961
Classification Concept 70: 0.899
Detection Concect 69: 0.084
AwA
Concept 10: 0.159
Concept 10: 0 0.358
Concept 67: 0.024
Concept 0.063
Concept 47: 0.011
Concept 47: 0.121
Concept 0.024
Concept 0.050
SUN
Concept 30: 4.2e-11
Concept 26: 0,051
Concept 27: 0.024
Concept 37: 0.037
Figure 11: Most important concepts for the Energy detector with respect to the predicted class "Buffalo". We demonstrate randomly sampled images that are predicted by the classifier into this class, We compare the top-4 important concepts to deseribe the DNN classifier (and Energy detector), ranked by the Shapley value based on classification completeness SHAP(T cx) (and detection completeness SHAP c;)). "Baseline" corresponds to the when the concepts learned with Ame Aucem corresponds to the concepts learned with Amss Anoen 0.1, Awep
Aup
whereas "Ours"
expanded set of concepts, and different concepts play major role in interpreting the classification and detection results.
D.2 MORE EXAMPLES OF OUR CONCEPT-BASED EXPLANATION
In Figure 12, we provide additional example of our concept-based explanation.
19

---

Under review as a conference paper at ICLR 2023
D (AmA)
Pidlo 000 000
0.75
Concept score
0.75
000 000
0.50
OCO (BUN
Concept
0.00
0.00
-0.25
C19
C35
holis
Concept score
0.73
Concegt score
0.755
000
0.50
Our (CS)
0.00
0.00
0.25
(a) class "Collie", Energy OOD detector. Images randomly selected from AwA test set and SUN.
(AmA))
Yah et al (C86)
Yeh
mofil 1930
moffl
Concapt score
0.75
Concapt score
0.75
1930
0:50
000
0.25
(Coa)
0.00
0.00
(AwA))
Ours (CB0)
Yeh et al.
Ours
Cancopt scara
0.75
Concept score
0.75
000
0.50
0.50
000 (Placos)
8:00
0:25
0.25
C35
C19
G11
C90
C37
(b) class "Dolphin", Energy OOD detector. Images randomly selected from AwA test set and P Places.
DD (Awsi)
Yeth al.
al. (C18)
1.00
Publle 000 -1 000
score
Concept score
0.75
OOO (Placee)
0.75
Meell
0.00
-0.25
C18
C20
C12
CM
00
(C34)
Yeh et al.
Ours
1.00
1.00
Conceept score
10 000
D
00..
0.50
Concept
0.50
(C1)
-0.25
-0.25
C38
C22
class "Dolphin", MSP OOD detector. Images randomly selected from AwA test and Places.
Figure 12: Concept- based explanations using concepts by Yeh et al. vs. ours. ID profile shows the concept-score patterns for normal ID images.
20

---

Under review as a conference paper at ICLR 2023
D.3 COUNTERFACTUAL ANALYSIS
To verify the important concepts identified by modified Shapley value, we perform counter- factual analysis, addressing the following ques- tion: if the OOD detector thought the input has different score for this concept, would the detec- tion result be different? As we do assume to have groundtruth annotation for concepts, we construct concept score profiles of detected- ID (or detected-OOD) inputs from held-out ID (or OOD) dataset, and refer to this as (or OOD) concept profile. With the guidance of ID and OOD concept profiles, we take intervention the concept scores of mis-detected inputs. Specifically, for ID) data mis-detected as OOD, we update their concept scores using ID profiles, and similarlly, for OOD data mis-detected as ID, their concept scores are updated with OOD profiles. The number of concepts to be intervened can be varied. As shown in Figure 13, with intervention more number of important concepts (ranked by SHAP(M,s c.)). we observe an improved performance of OOD detector concept world.
0.96
AUROC
0.84
Plccss SUN
Testures
10
20
25
Number concepts intervened
Figure 13: Performance of MSP with test-time inter- ventions concept scoress.
E DISCUSSION AND SOCIETAL IMPACT
Auxiliary OOD Dataset. limitation of our approach
requirement an auxiliary OOD dataset for concept learning, which could be hard to access in certain applications. To overcome that, research direction would be design generative models that simulate domain shifts or anomalous behavior and could create the auxiliary OOD dataset synthetically, allowing additional control on the extent distributional changes the resulting concepts could deal with (see Appendix for further diseussion).
Human Subject Study. Performing a human -uman-subject (or user) study would be the ultimate way to evaluate the effectiivneess of explanations, but remains largely unexplored even for in in-distribution classification tasks. We emphasize that designing such a usability test with OOD detectors would be
challenging due to the characteristics of the OOD detection task, compared to in- indistribution classification tasks. For in-distribution classifiers, users could potentially generate
hypotheses about what high-level concepts should attribute the class prediction, and compare their hypotheses to the provided explanations to determine the classifier's reliability. On the other hand, assessing the reliability of OOD detection involves checking whether given input belongs any of the natural distributions of concepts; this is essentially limited to whether users' mental models on such global distributions can be accurately probed via couple of presented local instances. We believe that designing thorough probing method for human interpretability on OOD detection would be an interesting yet challenging research quest by itself and our paper does not address that.
Societal Impact. Our work helps address the detection results of OOD detectors, giving practitioners the ability to explain the model's decision to invested parties. Our explanations can also be used to keep data point as understood mistake by the model rather than throwing it away without further analysis, which could help guide how to improve the OOD detector with respect to the concepts. However, this would also mean that more trust is put back into the human practitioner to not abuse the explanations or misrepresent them.
21


Reviews:
[{"review_id": "gvtV7yaJM87", "reviewer": "Reviewer_qdMm", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a method to interpret out-of-distribution detection using learnt high-level concepts.", "summary_of_the_review": "The paper poses an interesting question of how to use concepts to generate explanations from OOD detectors. While the evaluation and idea is adequate, it is hard for me to recommend accept without a convincing user study or extensive qualitative examples of why such an explanation makes sense. ", "strength_and_weaknesses": "Strengths\n- To my knowledge, this is the first work to study the notion of concept-based explanations for OOD detection.\n- The idea of LDA on the concept layer is clever and simple.\n- The proposed metrics are easy to understand. Detection completeness and concept separability naturally follow.\n\nWeaknesses\n- I would have expected more examples of what a concept-based explanation for an OOD detector means. Figure 1 is unconvincing. Please provide more qualitative evidence of the utility of concept-based explanations in the OOD setting.\n- While the authors acknowledge the lack of a human subject experiment, it is not clear from the prose alone if concept-based explanations are even necessary (let alone helpful) for OOD detectors. I, among others, am unsure why concepts for OOD detectors. How does it help? A carefully designed user study would right this. It would also warrant a method that would lead to higher adoption and impact down the line.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "c1SV3IZfUw", "reviewer": "Reviewer_v8DY", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes concept-based learning and an explanation for OoD Detection. The authors propose metrics to capture detection completeness and separability, which are used for the learning process. The evaluation of multiple datasets demonstrates the potential usefulness of the proposed techniques for OoD explanation.", "summary_of_the_review": "Overall, I like the idea of this paper and think it works on an important problem, which does not receive enough attention previously. This paper makes an early step in explaining OoD detectors in relatively general settings without the requirement of white box. However, I still have a few concerns posted in the weakness part, that hope the authors could address during the rebuttal phase and add relevant discussion in the next version.", "strength_and_weaknesses": "Strength:\n\n- Important topic and work on OoD detection explanations\n- Proposed techniques are general and could be applicable to various contexts, without the whitebox requirement\n- Feasible solution with promising results\n\nWeakness:\n\n- Unclear about the robustness of the learned “concepts”\n- Not very clear how to design the concept space or concept learning architecture\n- Although High level and simple concept explanations are possible, it is not clear how such explanations could be helpful to guide the OoD detection developers in an actionable and constructive way.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "L7O5Z3dKfOL", "reviewer": "Reviewer_YcEH", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper addresses the problem of explaining the decisions of an out-of-distribution (OOD) detector by using concept-based representations. In order to learn concept-based explanation, the authors propose two measures, namely the concept detection completeness and the concept separability, which they optimize during training of OOD detectors.  ", "summary_of_the_review": "The paper presents a novel idea, which is presented highlighting its relevance. The research is motivated and carefully constructed. The experimental analysis, although with several existing methods and on relevant datasets, brings to unfair statements in comparison with other methods, and lacks addressing strongly the validity of the proposed measures. Also, a more thorough analysis of the learned concepts should be provided: how are they relevant and what do they cover?\nOverall is a good work, but I would like to see the response of the authors to my questions and doubts.", "strength_and_weaknesses": "__Strenghts__\n- the problem addressed is relevant, and ground motivations for the research are discussed\n- the paper is well written and carefully constructed, easy-to-follow and clear\n- the new metrics are technically sound\n- experiments in combination with existing methods\n\n__Weaknesses__\n- In my opinion, the main concern/weakness is in the way the authors analyse and interpret the results: the authors stress the fact that existingmethods are not able to guarantee concept separability and detection completeness, which are metrics that they define in this work. On the contrary, the method based on learning concepts by optimizing these metrics is performing better. There is no surprise in these results, as previous methods are not optimized for the considered metrics. Thus, it seems to me that the experimentations (in the way it is presented) is unfair. \n\n- Another weakness is in the lack of details about the deployement of their approach within existing approaches: it seems a concept-based learning component (optimized with the proposed metrics) is deployed within intermediate layers, but no details about the 'where' in the network (which layer) it is used, and how it may change if deployed in different parts. \n\n- Connected to above, if this can be deployed in different parts of the network, I imagine that different concepts are learned, of different semantics. How does it relate with the 'high-level' concepts that the authors aim at learning? Also, how the 'high-level' is defined?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "1FbhrzVCGu", "reviewer": "Reviewer_H68v", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper tackles the problem of concept-based explanations for deep neural network OOD detectors. The authors build on Yeh et al. (2020)’s argument on concept completeness and propose to use two metrics: detection (concept) completeness and concept separability. Following such metrics, the authors then design an algorithm for learning such desired concept-based explanations. ", "summary_of_the_review": "This paper tackles the problem of concept-based explanations for deep neural network OOD detectors. The authors build heavily on Yeh et al. (2020), with the key differences that (1) they introduce an additional metric, i.e., concept separability, (2) instead of focusing on classification, they focus on OOD detection. Other aspects of the paper including the completeness scores and SHAP-based evaluation are very similar to Yeh et al. Given the limited technical merit, I would place this paper marginally below the threshold. ", "strength_and_weaknesses": "Strength \n\n+ The paper is very well written and well organized. It is a good read. \n\n+ Providing concept-based explanations to OOD detector is an interesting topic. \n\nWeaknesses\n\n- The notion of detection completeness is not entirely new. As shown in Definition 1 and 2, it builds on the classification completeness in Yeh et al. (2020). \n\n- The proposed method has a limited scope in that it only focuses on explaining OOD detector. \n\n- There are no direct comparisons between the proposed method and Yeh et al., which this paper heavily builds upon. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject