Automatic Concept Extraction for Concept Bottleneck-based Video Classification
Authors: Jeya Vikranth Jeyakumar
Luke Dickens
Yu-Hsi Cheng
Joseph Noor
Luis Antonio Garcia
Diego Ramirez Echavarria
Alessandra Russo
Lance M. Kaplan
Mani Srivastava
Venue: ICLR 2022 Submitted (2022)
Keywords: Explainable AI
Video Classification
Concept Extraction

Abstract: Recent efforts in interpretable deep learning models have shown that concept-based explanation methods achieve competitive accuracy with standard end-to-end models and enable reasoning and intervention about extracted high-level visual concepts from images, e.g., identifying the wing color and beak length for bird-species classification.  However, these concept bottleneck models rely on a domain expert providing a necessary and sufficient set of concepts--which is intractable for complex tasks such as video classification. For complex tasks, the labels and the relationship between visual elements span many frames, e.g., identifying a bird flying or catching prey--necessitating concepts with various levels of abstraction.  To this end, we present CoDEx, an automatic Concept Discovery and Extraction module that rigorously composes a necessary and sufficient set of concept abstractions for concept-based video classification. CoDEx identifies a rich set of complex concept abstractions from natural language explanations of videos--obviating the need to predefine the amorphous set of concepts. To demonstrate our method’s viability, we construct two new public datasets that combine existing complex video classification datasets with short, crowd-sourced natural language explanations for their labels. Our method elicits inherent complex concept abstractions in natural language to generalize concept-bottleneck methods to complex tasks.

Under review as a conference paper at ICLR 2022
AUTOMATIC CONCEPT EXTRACTION FOR CONCEPT BOTTLENECK-BASED VIDEO CLASSIFICATION
Anonymous authors Paper under double-blind review
ABSTRACT
Recent efforts in interpretable deep learning models have shown that concept- based explanation methods achieve competitive accuracy with standard end-to- end models and enable reasoning and intervention about extracted high-level visual concepts from images, e.g., identifying the wing color and beak length for bird-species classification. However, these concept bottleneck models rely on a domain expert providing a necessary and sufficient set of concepts- is intractable for complex tasks such as video classification. For complex tasks, the labels and the relationship between visual elements span many frames, c.g., identifying a bird flying or catching prey-necessitating concepts with various levels of abstraction. To this end, present CoDEX, automatic Concept Discovery and Extraction module that rigorously composes a necessary and sufficient set of concept abstractions for concept-based video classification. CoDEx identifies a rich set of complex concept abstractions from natural language explanations of coteitoibviating the need predefine the amorphous set of concepts. To demonstrate our method's viability, we construct two new public datasets that combine existing complex video classification datasets with short, crowd-sourced natural language explanations for their labels. Our method elicits inherent complex concept abstractions in natural language to generalize concept-bottleneee methods to complex tasks.
INTRODUCTION
Deep neural networks (DNNs) provide unparalleled performance when applied to application domains, including video classification and activity recognition. However, the inherent black-box nature of the DNNs inhibits the ability to explain the output decisions of model. While opaque decision- maaiinnnmaking may be sufficient for certain tasks, several critical and sensitive applications force model developers to face dilemma between selecting the best-performing solution one that is inherently explainable. For example, in the healtheare domain (Yeung et al. (2019). a life-or-death diagnosis compels the use automated prediction without justification is wholly
of the best performing model, yet accepting insufficient. Ideally, one could take ad advantage of the power of deep learning while still providing a sufficient understanding of why model is making particular decision, especially if the situation demands trust a decision that can have severe impacts.
To address the need for model interpretabilityy. researchers have sought to enable model intervention by leveraging concept bottleneck-based explanations. Unlike post hoc explanation methods-where techniques are used extract an explanation for a given input for inference by trained black-box model (Chakraborty et al. (2017): Jeyakumar et al. (2020)), concept bottleneck models are inherently approach to explaining a model inference based
interpretable and take a human meassonngginspiiee on an underlying set of concepts that define the decisions within an application. Thus far, prior works have focused on concept-basee explanation models for image (Kumar et al. (2009): : Koh et al.) (2020) and text classification (Murty et al. (2020). However, the concepts are assumed to be given priori by a domain expert-a process that may not result in a necessary and sufficient set of concepts. For instance, for bird species identification, an expert may provide two redundant concepts that are possibly correlated, such as wing color and beak color. More critically, prior works have considered simple concepts with the same level of abstraction, e.g., visual elements present in a single image. For more complex tasks such as video activity classification, label may span multiple frames. Thus, the composing set of concepts will have various levels of abstraction representing relationships of various

---

Under review as a conference paper at ICLR 2022
visual elements spanning multiple frames, e.g., a bird flapping its wings. Unlike the prior works, aim to exploit the complex abstractions inherent in natural language explanations to conceptualize such complex events.
Research Questions. In summary, this paper seeks to answer the following research questions:
How can machine automatically elicit the inherent complex concepts from natural language to construct necessary and sufficient set of concepts for video classification tasks?
Given that machine can extract such concepts, are they informative and meaningful enough be detected in videos by DNNs for downstream prediction tasks?
Are the machine extracted concepts perceived by humans as good explanations for the correct classifications?
Approach. This paper introduces an automatic concept extraction module for concept bottleneck- based video classification. The bottleneck architecture equips standard video classification model with an intermediate concept prediction layer that identifies concepts spanning multiple video frames. To compose the concepts that will be predicted by the model, we propose natural language processing (NLP) based automatic Concept Discovery and Extraction module, CoDEX, to extract a rich set of concepts from natural language explanations of video classification. NLP tools are leveraged to elicit inherent complex concept abstractions in natural language. CoDEX identifies and groups short textual fragments relating to events, thereby capturing the complex concepts from videos. Thus, we amortize the effort required to define and label the necessary and sufficient set of concepts. Moreover, we employ an attention mechanism to highlight and quantify which concepts are most important for given decision.
To demonstrate the efficacy of our approach, we construct two new MataasetsMML V2E (Video to Explanations) for baseball activity classification and MSR-V2B for video category classification- that combine complex video classification datasets with short, crowd-sourced natural language explanations for their corresponding labels. We first compare our model against the existing standard end-to-end deep-learning methods for video classification and show that our architecture provides additional benefits of inherently interpretable model with marginal impact on performance (less than 0.3% accuracy loss on classification tasks). A subsequent user study showed that the extracted concepts were perceived by humans as good explanations for the classification on both the MLB B-VEE and MSR-V2E datasets.
Contributions. We summarize our contributions follows.
We propose CoDEx, concept discovery and extraction module that leverages NLP tech- niques to automatically extract complex concept abstractions from crowd-sourced, natural language explanations for given video and label-obviating the need manually define necessary and sufficient set of concepts.
We evaluate our approach on complex video classification datasets and show that our model attains high concept accuracies while maintaining competitive task performance with standard end-to-ent video classification models.
We also augment the concept- based explanation architecture to include an attention mech- anism that highlights the importance of each concept for a given decision. We show that users prefer our concept extraction method over baseline methods to explain given label.
We construct two new public datasets, MLB-VVEE and MSR-V22 V2E, that combine complex video classification datasets with short, crowd-sourced natural language explanations and labels.
2 RELATED WORK
There is a wide array of works in explainable deep learning for various applications. This work focuses on the concepts-based explanations for video classification, and this section provides an overview of the existing literature for overlapping domains.
Concept-Ban Explanations for Images and Text. A number of existing works consider concept- bottleneck architectures where models are trained interact with high-level concepts. Generally, the
2

---

often results in concepts Under review as a conference paper at ICLR 2022
Concept Bottleneck Model
CoDEx Module
Concapts Prediction
[Extract Concepts)
NLP Framework
Eernetor dremsiia of
Tredated
Esplanation
Corpus (E)
Cleaning
Extraction
Completion
Grouping
Pruning
Vecto recttritation
Dateer Midect)
(M)
fanures
tatuse atek)
Concept Matrix C
Outputs
Figure 1: The overall pipeline showing the automatic concept extraction framework from natural language explanations and the concept bottleneck classification model training framework.
approaches are multi-task architectures, where the model first identifies human human-understee understandable set of concepts and then reasons about the identified concepts. Until now, the applications have been limited to static image and text upplications. [Khh al. (020)) used pre-labeled concepts provided by the dataset to train model that predicts the concepts, which is then used to predict the final classification. However, the caveat is that the concepts had to manually provided. Ghorbani et al.) (2019) and Yeh al. (2020) proposed approaches that automatically extract groups of pixels from the input image that represent meaningful concepts for the prediction. They were designed largely for image classification and extract concepts directly from the dataset. Kim al. (2018) propose post-hoc explanation method that returns the importance of user-defined concepts for a classification. In the mentioned works, the concepts have been limited to simple concepts and are not suited for complex tasks such as video classification where we have complex concepts that may span multiple frames with various levels of abstraction.
Explanations for Video Classification Other approaches have been considered to explain video classification and activity recognition. Chhttoopdhay et al. (2018) applied GradCAM and GradCAM++ to video classification, where for each frame, the important region of the frame to the model is highlighted heatmap. |Hiley al. (2020) extract both spatial and temporal explanations from input videos by highlighting the relevant pixels. However, these post-hoc techniques that focus on explaining black-box models, whereas our approach enables concept-botleeeck methods for video classification that are intended to be inherently interpretable and intervenable.
Video Captioning. In recent years, there is a large number of works (Pan et al. (2017) : Gao et al. (2017); Wang et al. (2018); Yan et al. (2019); Zhou et al. (2018); Chen & Jiang (202 D2: Yu et al. (0117) on video captioning. While they also employ natural language techniques, these works are tangential generating text explanations for classifications, since they are merely describing the video. Our model provides an explanation justifying the classification of the video. Similarly, the associated datasets such as MSR-VTT (Xu al. (016)) only have videos with ground truth captions that only describe the video without the context of classification-
that do not pertain to a classification. Semantic Concept Video Classification. The closest works to this paper is the body of work in semantic concept video classification (Fan et al. (2004) (007)), where the concepts are defined as salient objects that are visually distinguishable video components. The concepts in these works are simple objects detected in the videos and are not complex enough to capture the semantics of events that happen over multiple frames of the videos. These works typically used traditional SVM-based video classifiers. /Assari et al. (2014) represent video category based on the co-occurrences of the semantic concepts and classify based on the co-occurrences, but their method requires predefined set of concepts. Thus, we now present the methodology behind our automatic concept extraction for concept bottleneck video classification.
CONCEPT DISCOVERY AND BOTTLENECK VIDEO CLASSIFICATION
This work introduces CoDEx, automatic concept extraction method from natural language planations for concept-based video classification. Figure] depicts the overall pipeline, composed of ColEx and the concept bottleneck model. CoDEx extracts set of concepts
3

---

Under review as a conference paper at ICLR 2022
from natural language explanations that will comprise the bottleneck layer for the video classification model. We first formalize the overall problem and then provide the methodology for both modules.
Problem Formalization. We assume that have training dataset {(Ix. 4n)) D of videos In with a label I, E L, where is a predefined set of possible class labels for the video. Each video is represented
In = (fno- - sequence of frames where f represents frame t of video n. For each video In, we form a label-explanation pair (Ln, en), where en is (short) natural language document explaining the given label 4, If multiple annotators contribute to an explanation for video-label pair, (In, dn). then these are concatenated form En. The full set of pairs & {12x, ) is the explanation corpus. Thus, the design goals are:
FF where F is the set of video frames. Thus video
Concept Discovery and Extraction (CoDEX) Module: Given the explanation corpus, first produce an N K concept matrix, C, where the (n, k)th element if the mth explanation contains discovered concept k and 0 otherwise. We call the nih row Ca. the concept vector for video Xn. K is the total number discovered concepts.
Concept Bottleneck Model: Given a concept matrix, C, the second goal is to train concept bottleneck model such that for a given video I;, we predict a concept vector c,- which indicates the presence absence of concepts and their importance scores. The model then makes use of c, to make the final video classification.
Deplanation Corpus
Extracted
Iabel,
explaaationn
lobe,,
concepts
striee
axins] The bail was stile 2000.
strile
the batter cldd not sudag; the bail was the striie zono)
Sxl
(Tilbitee hit Bie hill Into theesterd and lorcle in Fudl territry.
Sxil
the butter bittheb it lorded faul tamittyy)
bail
7ix fittr daas's serng The lai was outiie the stile mon..
tell
the bxtter did nut swing, the bad was ouzside the strile mone)
Thevidoo load
botter ball It cought by the Bakdar)
out
the belter hit the teil and was ves by the felkeer
Completed
Groppdd concegts
concepts
Figure 2: Running example for all six stages of the discovery pipeline module. The left table is the explanation corpus, with highlighted fragments to be modified. The right table contains the discovered concepts. The detailed step-by-step modifications are provided in Appendix A.
3.1 CODEX: CONCEPT DISCOVERY AND EXTRACTION MODULE
We now describe CoDEx, that extracts concepts from the explanation corpus, E. The automatic extraction of the significant concepts done in 6 steps, as outlined in Fig. [1 These are: cleaning. extraction, grouping, completion, pruning, and vectorization, which produce the final concept matrix, C. Each of these steps are described below and illustrated with an example corpus depicted in Figure
Cleaning. We remove explanations associated with corrupted or unlabeled videos from the explana- tion corpus. In Figure 2 this phase would remove the fourth row with the "none" label.
Extraction. The objective of this phase is to identify sentence constituents relevant to explaining the label. These text fragments, short sequences of words that appear in the document, are referred to as raw concepts. To achieve this, the cleaned explanation corpus is tokenized then passed through pretrained constituency parser to recursively decompose the sentences. At each level of the constituency hierarchy. the text fragments are evaluated to determine whether they constitute a candidate concept. The rules for candidate raw concepts include the inclusion and exclusion rules below and follow the widely adopted Universal POS tag naming convention for token types (Petrov et al. (2012)). Every constituency parsed phrase that satisfies one of the two inclusion rules and not the exclusion rule is considered a candidate concept.
rule name
rule noun/ pronoun auxillary (optional) particle (optional) verb (optional) noun/ pronoun - auxiliary whose lemma is 'be' - any token
Inclusion 1.
Inclusion 2.
Exclusion subordinating conjunction
Table I: Inclusion and exclusion rules for candidate concepts.
4

---

Under review as a conference paper at ICLR 2022
After the extraction process is completed, have set of raw concepts, K, and each video is associated with subset of these raw concepts. An example of extracted raw concepts, AC, can be found in Appendix A.I
Completion. There are instances where the pretrained constituency parser will split sentences midway through a text fragment in one sentence that was kept whole in another. For instance, in Figure 2 the constituency parser splits the explamation for "foul" such that "the batter hit the ball" is incorrectly excluded from the raw concepts. To ensure that those concepts are captured, we perform a substring lookup of each raw concept through all documents of the explanation corpus and count an explanation as containing a raw concept if it contains the corresponding raw concept as a substring. This does not change the number of raw concepts identified but increases their frequency counts.
Grouping (similar raw concepts). When identical text fragments are identified in different expla- nations, they are counted directly as the same raw concept. However, we would ideally like to treat superficially different concepts the same if they essentially carry the same meaning, e.g., Figure hhighthh two different raw concepts that carry the same meaning and hence can be grouped. For this, difference between pairs of raw concepts and groups them together if they are similar enough. Our key contribution here is the distance metric used in clustering which is novel measure of meta-distance between raw concepts. This measures the difference between concepts based on two aspects of the raw concepts: their linguistic difference and their difference in terms of the label categories with which they are associated.
agglomerative clustering (Müllner (20II) approach that measures the degree of
We define meta-metric, d, as combining a linguistic distance, dwa (capturing linguistic difference) as well as a meta-metric, diabel (capturing the difference in the labels associated with each raw concept). More formally, for two raw concepts Ki, Kij E K our distance is linear combination:
d(Ks,, Ki)) = (Vs, V,) + Xdhabe ny)
where v; is sentence embedding for the text fragment of concept K; (e.g., based the BERT model (Devlin et al. 2019)), diet is a standard distance measure between vectors (e.g., cosine distance), diabel meta-distance which aims to capture the similarity between two label count vectors, and A is a hyperparameter controlling the relative importance between textual and label distance. The inclusion of a label distance helps distinguish between concepts that are superficially linguistically very simlar, but have very distinct meanings within the domain of interest. For instance, without the dianel, the concepts "the ball passed inside the strike zone" and "the ball passed outside the strike zone" will be grouped together though they are very different concepts as they have a very small dian We provide a more formal definition of the meta-metric diabel more formally and provide some intuition behind its construction in Appendix A.3 We also exclude concept groups which occur very rarely in the explanation corpus, with frequency less than some small threshold, t.
Pruning, Here, we seek compact subset of concepts that, together, capture a high degree of information about the label while maintaining interpretability. More formally, after grouping, we have a set of raw concepts K = [KI,..., k.J. and we seek some subset of maximally informative concepts K {Ki ..... Kjx C.
To see what is meant by maximally informative, consider randomly selected entry in the explanation corpus (1, e). We define binary random variable, Cj for each raw concept K.,, and for any concept set IC = {k.j:..,, Kij }. random vector Cx = [C1. Cx], such that C,- = if K, E e and 0 otherwise. Y is the random variable which takes label 1. We wish to choose the smallest subset of concepts such that the mutual information [MI[betweet chosen concepts, XC, and label, Y, given by I(Y; Cx), is greater than threshold fraction, 7 < 1 of the MI between the label and the complete concept vector, [(; CC)) That is to say we wish to find XC which satisfies:
I(Y; Cx) 2 71(x
(2)
and where there is no subset K" C K with [K'| < |A] which also satisfies Equation equation 2 In
practice, this is infeasible as the problem is combinatorial. However, we note that f(K) = T(Y Cx)
is monotone submodular set function of K. Given this, if we recursively construct set of size K. by greedily adding single concepts that most improve the MI, the resulting set will be at least
We use the standard definition of mutual information (MT) for discrete random variables MacKay 2003
5

---

Under review as a conference paper at ICLR 2022
good the most informative set of size K (Nemhauser al. (1978)). Therefore, we guarantee highly-infon informaitfe set KC by iteratively adding concepts to those previously selected, greedy with respect to the MI, until we have set that satisfies Equation
Vectorization. Each concept Kj. E IC is given a unique index & E [L....,K] K). and each data- point, Zn is associated with concept vector Cn (Ca1,..., ,.lg.......k,, where Cak 1 if Kj,, en and O otherwise, indicating the presence or absence of the kthh concept in the nth explanation. The collection of all the concept vectors gives an N K concept matrix, C.
3.2 CONCEPT BOTTLENECK MODEL
We use the videos, the extracted concepts from CoDEx, and the labels to train an interpretable concept-bottleneck model to predict the activity and the corresponding concepts. Figure shows the overview of our bottleneck architecture. The activity label, the concepts, and the corresponding concept scores are the outputs of the interpretable model and are indicated by dotted arrows in Figure I
Our bottleneck model architecture is based on the standard end-to-end video classification models where convolutional neural network-based feature extractors pretrained the Imagenet dataset Deng et al. (2009) to extract the spatial features from the videos. The features are then passed through temporal layers that can capture features across multiple frames which in turn is bottle-necked to predict the concepts. Lastly, we deploy an additive attention module (Bahdanau et al. (2014) that gives the concept score a, indicating the importance of every concept to the classification. The attention module also improves the interpretability of the the bottleneck model by indicating the key concepts for classification and this is evaluated in section More details regarding the model architecture and hyper-parameters parameters are in the Appendix A.5
Model Joss function. The entire bottleneck classification model is trained in an end-to-end manner. Since the concepts are represented as binary vectors, we use sigmoid activation on the concept bottleneck layer and binary categorical loss function as the concept loss. The final layer of the classifier has softmax activations and categorical cross-entropy as the classification loss function. Thus, the overall loss function of the model is the sum of concept loss and the classification loss. The hyperparameter 3 controls the tradcoff between concept loss, Lc. versus classification loss, Ly as shown in equation The full expansion the equation is located Appendix A.5
1 N Loss(L) = N ==1
Σ (Ly, + Lc.))
where
B > 0
Testing phase. Given an input test video, the model provides with the activity prediction (label of the video), concept vector indicating the relevant concepts that induced this classification and the concept importance score for each concept. By retrieving the phrase representing the concepts present in the video, the result obtained is human-understandable explanation of the classification.
4 IMPLEMENTATION
To demonstrate our automatic concept extraction method, we construct two new datasets MLB-V2E (Video to Explanations) and MSR- MSR-V2E, which combines short video clips with crowd-sourced classification labels and corresponding natural language explanations. For both datasets, we obtained a video activity label and natural language explanations for that label by crowd-sourcing on Amazon Mechanical Turk and used unrestricted text explanations to extract concepts automatically. For IRB exemption and compensation information, please refer to the Ethics Statement.
MLB-V2E Dataset: We used a subset of the MLB-YA Youtuuee video activity classification dataset introduced by Piergiovanni & Ryoo (2018)-whie
had segmented video clips containing the five primary activities in baseball: strike, ball, foul, out, in play. We preprocessed the dataset and extracted 2000 segmented video clips where each video was 12 seconds long, 224 224 in resolution, and recorded at 30 fps. To ensure that the quality of explanations is good, we screened over 450 participants. Based on their baseball knowledge, 150 participants were qualified to provide the natural language text explanations for our video clips. We have included a sample of our screening survey, the primary survey, and the explanations collected in the supplementary materials.

---

Under review as a conference paper at ICLR 2022
Number of Concepts after each Phase
Dataset
Extraction
Completion
Grouping
Pruning
MLB-V2E
1885
1885
225
80
AIRR-VE
1678
1678
104
Table The number of concepts extracted by the Concept Discovery module from the explanation corpus after every phase.
0.68
0.80
ArquracyF1 Spore
AccuracyF1150 1--oore
0.58
0.66
D66
0.54
0.02
0.52
Accuany Fl Scom
0.50
FirScore
0.60
150
100
Number of Concepts
Number Concepts
(a) MLB-VZE Dataset
(b) MSR-V2E Dataset
Figure 3: The number of concepts versus performance trade-off for the (a) the MLB-V2E dataset and (b) the MSR-V ERR-V2E dataset.
For this dataset, we used 2020 video clips from the MSR VTT dataset introduced by Xu et al. (2016). The MSR-VTT dataset has general videos from everyday life and descriptions of these videos associated with them. Each video clip is between 10-30 seconds long, and approximately 200 participants provided the labels and explanations to construct the MSR- V2E dataset. The videos are classified into ten categories: Automobiles, Cooking, Beauty and Fashion, News, Science and Technology, Eating, Playing Sports, Music, Animals, and Family (more details in Appendix [A. Training: All our models were trained on x Titan GTX GPUs using Adam optimizer. A summary of our entire model architecture and trained model is provided in the supplementary materials.
MSR-V2L Dataset:
RESULTS
Number of extracted concepts. Table 2 shows that the system extracted 80 concepts and 62 concepts from the explanation corpus of MLB-V2E and MSR-V2E respectively. The number of concepts remaining after the pruning phase is determined by the cumulative Mutual Information(MI) threshold. To identify the best threshold, we plotted the number of concepts at different thresholds versus performance of the model as shown in Figure 3 We found that the task classification performance did not increase after a certain number of concepts and that optimal spot for the number of concepts corresponded to 90% Mutual Information (Appendix A.6).
Comparing concept-bottleneeck models to baselines. We adopt model architectures and hyperpa- rameters from standard well well-performing approaches that fall under 3 categories: 1) without concept bottleneck, 2) with concept bottleneck, 3) with concept bottleneck and attention. We compared the performance of models with the bottleneck layer with standard video classification models without concept bottleneck layer. We find that, though the latent space constrained to the limited set of concepts extracted from the explanation corpus, concept models performed as well as the uncon-
Feature Extractor
Task Classification
Concepts
Dataset
Model Type
Accuracy(%)
F1-score
AUC
Standard Bottleneck
68.46 1: 1.27
0.68 1 0.011
MLB-V2E
Inception
68.16 +: 1.12
0.68
0.004
0.85 1 0.003
Bottleneck + Attn.
68.38 +: 1.34
0.68 + 0.004
0.88 1 0.001
Standard Bottleneck Bottleneck Attn.
61.79 + 1.42
0.60 + 0.012
MSR-VD V2E
Inception V3
61.42 + 1.18
0.60 + 0.013
0.83 + 0.006 0.86 + 0.004
61.68 + 1.23
0.60 + 0.009
Table 3: Performance of Models. The full table be found in Appendix A.7
7

---

Under review as a conference paper at ICLR 2022
Dataset
Concepts Extraction
Task Accuracy($0
Task Fl-score
Concept AUC
CoDEx
68.38
0.6802
0.8801
w/o Extraction
63.47
0,5823
0.8185
MLB-V2E
w/o Grouping
67.80
0,6772
0,8122
w/o Pruning
68.36
0,6802
0.8419
w/o Grouping and Pruning
65.29
0.6526
0.7821
CoDEx
61.68
0.6010
0,8600
w/o Extraction
58.02
0.5214
0.7830
MSR-V22 -2R-V2E
w/o Grouping
59.31
0.5745
0.7888
w/o Pruning
61.68
0.6010
0.8131
w/o Grouping and Pruning
54.70
0.5178
0.7467
Table 4: Ablation Studies: Shows the effect of each step in CoDEx on model performance
strained models, on both datasets. We also find that the addition of the attention layer improves the concept prediction of the models. T shows that concept bottleneck models achieved comparable task accuracy to standard black-box models on both tasks, despite the bottleneck constraint while achieving high concept prediction performance. Appendix A. shows the performance with other feature extractors.
Predicted Cless: YOur
Predicted Class: Beaut Fashion
The bater HiPe ball
to aaa anow
Thtt was apptingg makoup face
eegtenigg beauty product
Othrr
Othert
Concep
Scar
Canoic Soae
MUB-VZE Dataset
MSR-VZE Dataset
Figure 4: Explanation offered by the model indicating the predicted class concepts present and their corresponding scores for (a) the MLB-V2E dataset (b) the MSR V2E dataset.
Ablation Study CoDEx. We performed an ablation study to highlight the impact of CoDEx's components. For sentence-level concept extraction, we replaced the extraction phase with word-level concepts extracted from the explanation corpus. We also evaluated how removing the grouping and pruning phases would impact performance. shole the results of our study. We find that using word-level concepts significantly reduced the performance of predicting the task and the concepts. The grouping and pruning stages had greater impact the concept prediction performance that affected the explainability of the model.
Concept scores for interprefability. Not only does the attention module increase performance in concept prediction, but it also improves the explainability the bottleneck model by providing an importance score for the concepts. Figure 4 shows the explanation from the concept bottleneck model with attention on a test sample from the two datasets. More examples can be found in Appendix 10 The title shows the classification label, the y-axis indicates the top-3 concepts predicted as present in the video clip, and the x-axis corresponds to the concept score. Others refers to the sum of the importance of all the remaining concepts.
Human study to evaluate concepts' explainability. We performed a Mechanical Turk study to evaluate the explainability of our extracted complex concepts to the end-users. The participants were asked to select from four different options (presented in random) of what they consider to be the best possible Explanation for the classification of given video. The four options are: Complex concepts predicted models without attention, Complex concepts predicted by models with attention, Concepts of random video not belonging to the same predicted class and a Random set of 2-5 concepts from the set of the most frequent concepts. The methodology of this study was inspired by CCang et al. (2009)'s 's paper. Figure (rsuree5preeentt the aggregated results the Mechanical Turk study. The complex concepts predicted by the concept bottleneck model with attention was considered the preferred explanation by 68% and 57% of the responses in the MLB- V2E and MSR- V2E datasets respectively followed by the concepts bottleneck models without attention in 20% and 28% of the responses for
8

---

Under review as a conference paper at ICLR 2022
80
70
60
Percentage (%)
Percentogn (%)
40
30
30
10
10
Pandemrci
Beaken Coraple
Senteral
Canopio
Survey Options
Survey Options
(4)
Figure 5: Survey responses with 95% bootstrap confidence interval for the two datasets
the two datasets. The presented confidence intervals are calculated using the bootstrap method as described by DiCiccii & Efron 1996) for 95% confidence.
DISCUSSION
Annotation effort. Although CoDEX requires collecting large explanation corpus, prior work requires two studies: one study to identify the set essential concepts, and second study to annotate the videos with the essential concepts-whereas CoDEX only requires single study, Further, the vocabulary of concepts large, the annotation process would be inconvenient for the user. Moreover, natural language explanations can express richer compositions of concepts rather than simply identifying the presence or absence of an individual concept. Thus, CoDEX's 'x' annotation efforts more expressive, less expensive and less cumbersome than prior works.
Representative Concept. Our concept extraction method selects the most frequent concept in grouped cluster the representative concept. In general, the most frequent concept suffices to explain particular component of the complex activity. However, there were some instances where most frequent concept would have a specific terminology rather than a general term, e.g., "left fielder" is subclass of "outfielder." Future work can strive towards generating the representative concept for cluster, as opposed to opting for the most frequent or popular phrasing.
Preserving Spatial-temporal Semantics. Our model's output explanation currently provides of activated concepts along with their score. However, they do not capture the spatial and temporal relationships between concepts. Some rich concepts implicitly embed spatial and temporal properties, e.g., "the batter hit the ball on the ground" implies the following sequence: a batter swung at ball, made contact with the ball, and the ball landed on the ground. However, if the generated set of concepts is limited to less informative concepts, e.g., "the batter," the spatial and temporal ordering of concepts matters. Future work can generalize the architecture to generate concept- based natural language explanations that explicitly preserve spatoal-temporal semantics.
Neural-symbolic Reasoning. Our model's reasoning layers are inherently black-box in nature, i.e., the concept vectors are fed into a fully connected network. To further bolster human-maching teaming and interpretability, the final classification model can be replaced with rule-based model-analogous to prior works that fuse deep learning inferences with symbolic reasoning layers for complex event detection (Xing et al. (200);; Vilamala et al. (2019).
CONCLUSION
The remarkable performance of deep neural networks is only limited by the stark limitation in clearly explaining their inner workings. While researchers have introduced feature highlighting explanation techniques provide insight into these black-box models, concept-botllenck models offer promising new approach to explanation by decomposing application tasks into a set of underlying concepts. We build upon concept-based explanations by introducing an automatic concept extraction module, CoDEx, to a general concept bottleneck architecture for identifying, training, and explaining video classification tasks. In coalescing concept definitions across crowd-sourced explanations, CoDEx amortizes the expertise of concept definition while removing the burden from the model developer. We also show that our method provides reasonable explanations classification without compromising performance compared to standard end-fo-end video classification models.

---

Under review as a conference paper at ICLR 2022
ETHICS STATEMENT
IRB Exemption and Compensation. This research study has been certified as exempt from review by the IRB and the participants were compensated at rate of 15 USD per hour for a total of 920.36 USD spent.
Dataset privacy. There personally identifiable information collected at anytime during the turk study. The responses provided by the mechanical turkers that are present in the dataset are completely anonymous.
REPRODUCIBILITY STATEMENT
The entire code with detailed comments are provided in the supplementary materials. The model architectures and hyper- parameterssss used are discussed in Appendix A.5 All the plots and graphs can be obtained by running the code without modifications.
REFERENCES
Shayan Modiri Assari, Amir Roshan Zamir, and Mubarak Shah. Video classification using semantic concept co-occurrences. In 2014 IEEE Conference Computer Vision and Pattern Recognition, pp. 2529-2536. 2536. IEEE, 2014.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint
2014.
Supriyo Chakraborty, Richard Tomsett, Ramya Raghavendra, Daniel Harborne, Moustafa Alzan- tot, Federico Cerutti, Mani Srivastava, Alun Preece, Simon Julier, Raghuveer M Rao, et al. Interpretability of deep learning models: survey of results. In 2017 IEEE smartworld, ubiq- uitous intelligence & computing, advanced & trusted computed, scalable computing & com- munications, cloud & big data computing, Internet of people and smart city innovation (smart- pp. 1-6. IEEE, 2017.
Jonathan Chang, Sean Gerrish, Chong Wang, Jordan L
and David M Blei. Reading tea leaves: How humans interpret topic models. In Advances in neural information processing systems, pp. 288-296, 2009.
Aditya Chattopadhay, Anirban Sarkar, Prantik Howlader, and Vineeth N Balasubramanian. Grad- cam++: Generalized gradient-based visual explanations for deep convolutional networks. In 2018 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 839-847. IEEE, 2018.
Shaoxiang Chen and Yu-Gang Jiang. Towards bridging event captioner and sentence localizer for weakly supervised dense event captioning. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 8425-8435 2021.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 JEEE conference on computer vision and pattern recognition, pp. 248-255. Ieee, 2009.
Jacob Devlin, Ming-Wai Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding, 2019.
Thomas DiCiccio and Bradley Efron. Bootstrap confidence intervals. Statistical science, 1 11(3): 189-228, 1996.
Jianping Fan, Hangzai Luo, Jing Xiao, and Lide Wu. Semantic video classification and feature subset selection under context and concept uncertainty. Proceedings of the 2004 Joint ACM/IEEE Conference on Digital Libraries, 2004., pp. 192-201. IEEE, 2004.
Jianping Fan, Hangzai Luo. Yuli Gao, and Ramesh Jain. Incorporating concept ontology for hierar- chical video classification, annotation, and visualization. IEEE Transactions on Multimedia, 9(5): 939-957, 2007.
10

---

Under review as a conference paper at ICLR 2022
Lianli Gao, Zhao Guo, Hanwang Zhang, Xing Xu, and Heng Tao Shen. Video captioning with attention-based Istm and semantic consistency. IEEE Transactions on Multimedia, 19(9):2045 2055, 2017.
Amirata Ghorbani, James Wexler, James Zou, and Been Kim. Towards automatic concept-based explanations. arXiv preprint arXiv:1902.031 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Liam Hiley, Alun Preece, Yulia Hicks, Supriyo Chakraborty, Prudhvi Gurram, and Richard Tomsett. Explaining motion relevance for activity recognition in video deep learning models. arXiv preprint arXiv:2003. /4285, 2020.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735- 1780, 1997.
Jeya Vikranth Jeyakumar, Joseph Noor, Yu-Hsi Cheng, Luis Garcia, and Mani Srivastava. How can i explain this you? an empirical study of deep neural network explanation methods. Advances in Neural Information Processing Systems, 33, 2020.
Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, et al. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). In International conference on machine learning, pp. 2668-2677. PMLR, 2018.
Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. Concept bottleneck models. In International Conference on Machine Learning, pp. 9338-5348. PMLR, 2020.
Neeraj Kumar, Alexander Berg, Peter N Belhumeur, and Shree Nayar. Attribute and simile classifiers for face verification. In 2009 IEEE 12th international conference computer vision, pp. 365-372. IEEE, 2009.
Colin Lea, Michael D Flynn, Rene Vidal, Austin Reiter, and Gregory D Hager. Temporal convolutional networks for action segmentation and detection. In proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 156-165, 2017. 2017.
David J. C. MacKay, Information Theory, Inference, and Leariing Algorithms. Cambridge University Press, 2003.
Daniel Müllner. Modern hierarchical, agglomerative clustering algorithms. arXiv preprint arXiv:1109.2377, 2011.
Shikhar Murty, Pang Wei Koh, and Percy Liang. Expbert: Representation engineering with natural language explanations. arXiv preprint arXiv:2005.01992 2020.
G. L. Nemhauser, L. A. Wolsey, and M. L. Fisher. An analysis of approximations for maximizing submodular functions Mathematical PropiicaaaPooraaaa Programming. 14:265-294, 1978.
Yingwei Pan, Ting Yao, Houqiang Li, and Tao Mei. Video captioning with transferred semantic attributes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017.
Slav Petrov, Dipanjan Das, and Ryan McDonald. A universal part-of-speech tagset. In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC' 12), pp. 2089-2096, 2012.
AJ Piergiovanni and Michael S. Ryoo. Fine-grained activity recognition in baseball videos. In CVPR Workshop on Computer Vision in Sports, 2018.
Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese nett-networks. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 2019. URL htt.:tp::/// arxiv. orq/ abs /1908
1 10084
11

---

Under review as a conference paper at ICLR 2022
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, distilled version of bert: smaller, faster, cheaper and lighter. ArXiv, abs/1910.01108,, 2019.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2818-2 18-2882, 2016.
Marc Roig Vilamala, Liam Hiley, Yulia Hicks, Alun Preece, and Federico Cerutti. A pilot study on detecting violence videos fusing proxy models. In 2019 22th International Conference on Information Fusion (FUSION), pp. 1-8. IEEE, 2019.
Bairui Wang, Lin Ma, Wei Zhang, and Wei Liu. Reconstruction network for video captioning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7622-7631, 2018.
Tianwei Xing, Luis Garcia, Mare Roig Vilamala, Federico Cerutti, Lance Kaplan, Alun Preece, and Mani Srivastava. Neuroplex: learning to detect complex events in sensor networks through knowledge injection. In Proceedings of the 18th Conference on Embedded Networked Sensor Systems, pp. 489-502, 2020.
Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging video and language. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2288-529,, 2016.
Chenggang Yan, Yunbin Tu, Xingzheng Wang. Yongbing Zhang, Xinhong Hao, Yongdong Zhang, and Qionghai Dai. Stat: Spatial-tempont attention mechanism for video captioning. IEEE transactions on multimedia, 22(1):229- 241, 2019.
ChihhKuan Yeh, Been Kim, Sercan Arik, Chun-LLiang Li, Tomas Pfister, and Pradeep Ravikumar. completeness-aware concept-basee based explanations in deep neural networks. Advances in Neural Information Processing Systems, 33, 2020.
Serena Yeung, Francesca Rinaldo, Jeffrey Jopling, Bingbin Liu, Rishab Mehra, N Lance Downing, Michelle Guo, Gabriel M Bianconi, Alexandre Alahi, Julia Lee, et al. A computer vision system for deep learning-basee detection of patient mobilization activities in the icu. NPJ digital medicine, 2(1):1-5, 2019.
Youngjae Yu, Hyungjin Ko, Jongwook Choi, and Gunhee Kim. End-to-end concept word detection for video captioning, retrieval, and question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3165-3173, 2017.
Luowei Zhou, Yingbo Zhou, Jason Corso, Richard Socher, and Caiming Xiong. End-to-end dense video captioning with masked transformer. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8739-8748, 2018
A APPENDIX
A.I RUNNING EXAMPLE DESCRIBE THE CONCEPT DISCOVERY PIPELINE
Example (A simple Explanation corpus). Consider using baseball domain with few meaningful labels, null label {strike, ball. foul, out, none) and five entries the explanation corpus E:
id. label, Ln
explanation, on
strike
The batter did not swing. The ball was in the strike zone.
foul
the batter hit the ball into the stands and it landed in foud territory
ball
The hitter didn't swing. The ball was outside the strike zone.
none
The video did not load.
out
the batter hit the ball and was caught by the fielder
Example (After Cleaning Phase). The none label entry is removed after the Cleaning phase
12

---

Under review as a conference paper at ICLR 2022
id,
n
label, in
explanation, En
1
strike
The batter did not swing. The ball was in the strike zone.
2
foul
the batter hit the ball into the stands and it landed in foul territory.
3
ball
The hitter didn't i swing. The ball was outside the strike zone.
4
out
the batter hit the ball and it was caught by the fielder
Example (After Extraction Phase). The raw concepts are extracted based on the defined rules discussed in Table correppoddnng each explanation with the help of a pre-trained constituency parser.
id.
label, Ln
raw concepts. K.
strike
The batter did not swing. The ball was in the strike zone
2
foul
the ball into the stands, it landed in foul territory
ball
The hitter didn't swing, The ball was outside the strike zone the batter hit the ball, it was caught by the fielder
4
out
Example (After Completion Phase). The concept 'the balter hit the ball' not extracted by the Extraction phase for the Id 2 explanation in this corpus. This missing concept is retrieved through the sub-string matching.
id, n
label, In
raw concepts, K
1
strike
The batter did not swing, The ball was in the strike zone
2
foul
the batter hit the ball, the ball into the stands, it landed in foul territory
3
ball The hitter didn't swing, The ball outside the strike zone
4
out
the batter hit the ball, it was caught by the fielder
Example (After Grouping Phase). We show the grouped concepts for this example corpus Concept-index, Concept groups
[The batter did not swing, The hitter didn't 't swing] [the batter hit the ball, the batter hit the ball] [The ball was in the strike zone] [The ball into the stands] lnt landed foul territory] [The ball was outside the strike zone] was caught by the fielder]
2
3
4
5
6
7
Example (After Pruning Phase). The concept 'the ball into the stands' of Concept-Inder from previous table was not contributing much and hence was pruned.
Concept-index,
Concept groups
1
[The batter did not swing, The hitter didn't 't swing] [the batter hit the ball, the batter hit the ball] [The ball was in the strike zone] In landed in foul territory] [The ball was outside the strike zone] [it was caught by the fielder]
2
3
4
5
6
Example (After Vectorization), After pruning, each sample is mapped to their corresponding concept vector. The value of concept vector index is 1 if e, has the concept with index i, else, it's 0. The Matrix containing all the concept vectors is called the Concept Matric, C
id.
label, In
Concept Vector, Ca.
strike
[1,0,1,0.0.0] [0,1,0,1,0,0]
foul
ball
[1,0,0.0,1.0]
4 out
[0.1,0,0.0,1]
13

---

Under review as a conference paper at ICLR 2022
swung
The
swung
the
NO
VP
NP
vn
The betler
swarg and missed
The ball
was in the strike-zone
DT
NN
VBD
The
batter
swung
and
missed
The
ball
was
the zone
swung
missed
the sirike-zone
the
strike
zone
Figure 6: Constituency tree for explanation e1 from Example A.1
A.2 EXAMPLE CONSTITUENCY TREE
The explanation n from Example A. is is decomposed into the constituency tree shown in Figure 6 The parser gives a hierarchy of constituents. Our method traverses through this tree and selects the constituents that satisfy the rules discussed Table T important to note that rules can added or deleted at this step based on the requirements.
A.3 META-DISTANCE FOR LABEL BASED PROXIMITY
At the end of the Completion Phase we define a count for each raw concept, E K, given by Mc. And for each label category 1 E L, we define label count for raw concept K; ma where i is the index of the concept. These count the presence of raw concepts explanations, and Eiec my My. Finally we group together raw concept K, 'ss label counts into label count vector
m; [m. 1m...... .., m
Now we describe the meta-metric dlabel used the Grouping phase more formally and provide some intuition behind its construction. Consider that we have two raw concepts Kg, Kg, Ki E K and label count vectors m, and my. We next assume that vector m, constitutes M, i.idd draws from categorical distribution with unknown parameters μ4 = (μu)) t= where μ4 is the probability that randomly selected occurrence of raw concept i belongs to an entry in the explanation corpus with label category 1. Our label distance diabel is the evidence ratio between the count vectors, m, and my, being drawn from independent categorical distributions (model Mindp) versus them being drawn from the same distribution (model Meamb.. More precisely,
p(m, my Mindp)
diabel m m,) p(m;, my Meamb)
Note that this not a true distance between count vectors as two identical count vectors not have a distance of zero. Nonetheless, it satisfies the other requirements of metrie: non-negativity. symmetry and the triangle inequality, and two vectors that are more (less) likely to come from the same multinomial will have distance less (more) than
To evaluate the label distance we must calculate the evidence for various categorical samples given the model p(m μa, M) For simplicity, assume total count M is known and define Dirichlet prior ((pu)a1) where 1 is the vector all 1s (this makes the simplifying assumption that the prior is symmetric). The evidence for m then:
p(m|a)= (aal))>=<(mmpe p(mmis, M)p(44aa1)d
The label meta-metric is the evidence ratio given by:
(m, m,) = p(m; (imm:aa)pp (a)p(mm(aa))
duabel (m,
p( (mm; + m,la)
14

---

Under review as a conference paper at ICLR 2022
For computational efficiency (and since it did not appear to affect results measurably) approximation for p(m]at a in our calculation dj.
We first evaluate the expected parameter of the posterior distribution given count vector m, namely
j = E ml
then evaluate the evidence for conditioned ju. i...
p(m (
II k=1
K
my + a
mal
M, + Ka
We then calculate log dlabel (m,, my) and exponentiate to improve precision. After grouping, the raw concept within cluster with highest frequency is identified as the representative concept of that cluster.
A.4
LANGUAGE MODELS
A.4.1 CONCEPT EXTRACTION
After obtaining the free form textual explanations for both, we first cleaned them by removing explana- tions associated with corrupted video files and the videos which were labelled incorrectly. We then con- sidered three different Spacy's 's pretrained constituency parsers: en_core__t rn__cor_teeb.b. e__eee__lg en_core_well sie___ee_mddd en_core jv__ore_weeb web sm parse the explanations and extract raw concepts based the rules discussed in section 3.1 We found that, the parser en en_core_____ _ore__eb_________________ee_ee__ llg was more accurate in identifying the constituents and resulted in better concept extraction.
A.4.2 CONCEPT GROUPING
For text distance embedded the raw concepts with sentence encoder and we exper- imented with two models: paraphrase-diste trobertt-baseeee
(distil) [Sanh et al. (stsb) Reimers Gurevych (2019) both using the python library, and evaluated a variety of distance metrics within the resulting 768 dimensional space, including: Chebyshev (infinity norm), manhattan, Euclidean and cosine distances.
(2019) and stsb-roberta-bas sentence_trance
And to cluster the semantically similar concepts together using agglomerative clustering Müllne (201T), we evaluated a variety of distance metrics within the resulting 768 (68-dimensional space, including: Our proposed meta-distance metric, Chebyshev (infinity norm), manhattan, Euclidean and cosine distances. To select hyperparameters, including: choice of sentence embedding model, distance metric for sentence embeddings, prior a for label distance, and relative importance factor performed grid search and selected the values that resulted in well-formed clusters. Based on our experiments (Provided in supplementary materials), we found that stsh encoder with our proposed meta-distance metric resulted in the best grouping of concepts. Note: dient be either cosine or manhattan distance as they gave similar clusters.
After clustering, set the frequency occurrence threshold of 3 and removed the rare concept groups which occurred less than this threshold. Then, pruning was done using 90% of mutual information score discussed in section 3.1 which resulted in 80 significant concepts for our MLB-V dataset and 62 concepts for MSR- V2E dataset as shown in Table and Figure Therefore, each video was associated with binary concept vector of shape [1 x k] where is the number of concepts, indicating the presence and absence of each concept.
A.5 THE CLASSIHCATION MODELS
We considered three different feature extractors Resnet 50v2 He et al. (2016). Resnet 101 He al. (2016) and Inception Szegedy et al. (2016) models and pretrained the Imagenet dataset to extract features from each frame of our video clips. We excluded the final classification layer from these models and did global maxpool across the width and height such that get 2048 size feature vector for every frame. We then concatenate the features together, resulting in a [2048 x 360] feature matrix for every video where 360 is the number of frames per video.
15

---

Under review as a conference paper at ICLR 2022
0.50 propertion of
0.80 prggrrion of
Sei 24 canapla
0.4
Humbee Concapte
Number
MARZE Datowet
(1) MSR-13E Detsset
Figure 7: Selecting concepts based on the Mutual Information. (a) V2E dataset (b)MSR-V2E
Dataset
For the Temporal Layer, we considered both temporal convolution Lea et al. (2017) and LSTM Hochre- iter & Schmidhuber (1997) based architectures which good extracting temporal features and found that Temporal CNNs outperformed LSTM by significant amount. And the Bottleneck Layer is dense layer with & neurons and hence the output is vector of shape [1 k] where k is the number of significant concepts. We introduced an attention layer in the concept-botrlenee for each concept. The final fully connected layer is implemented with L.
model
that gives the concept neurons (L classes) which predicts the class from the video.
Model Loss function Assuming s the feature vectors obtained from the Feature extractors
Loss(L) N
(Ly, + x Lcn)
(4)
1
m
B
- log(fo (1 cs log((1 - (sk)))) - Σ
log fs($))
n
i=1
k=1
j=1
Ex
where
(8(
and
[s({,)
B>0
Ite
mm Lj==1 e*
A.6 SELECTING CONCEPTS BASED THE MUTUAL INFORMATION(MI)
Figure 7 shows the plot between cumulative MI and the number of concepts after pruning. As discussed in Section | the sweet spot for the number of concepts corresponded to 90% of the cumulative MI beyond which there was no gain in classification performance as increased the number of concepts.
A.7 PERFORMANCE OF MODELS
Table shows the performance of all the models with different feature extractors. Each model was trained thrice and the mean and standard deviations are reported. We find that models with Inception V3 the feature extractor performed the best. Adding attention mechanism greatly improved the performance of concepts prediction and also achieved higher accuracies than the concept bottleneck models without attention.
A.8 THE RELATIONSHIP BETWEEN CONCEPTS AND THE CLASSIFICATION TASK
To understand the relationship between the extracted the concepts and the task classification, we compared the performance of the Concept Bottleneck models with a) MLP classifier b)Linear Classifier as the final classification layers. The results showed that there was approximately 2% drop in classification performance when using Linear Classifier instead of an MLP. This indicates that, the classification task is not a simple linear combination the extracted concepts and the composition of concepts is important.
16

---

Under review as a conference paper at ICLR 2022
Feature Extractor
Task Classification
Concepts
Dataset
Model Type
Accuracy($)
Fl-score
AUC
Standard
67.92 + 0.78
0.68 + 0.003
Resnet 50V2
Bottleneck Bottleneck + Attn.
67.83 0.74
0.68 0.001
0.85 0.005
67.96 + 0.65
0.68 + 0.002
0.88 + 0.002
Standard Bottleneck Bottleneck + Attn.
68.18 + 0.88
0.68 + 0.005
MLB-V2 B--E
Resnet 101V2
68.01 1.02
0.68 = 0.013
0.85 0.004
68.26 + 1.12
0.68 + 0.009
0.88 + 0.000
Standard Bottleneck Bottleneck + Attn.
68.46 + 1.27
0.68 + 0.011
Inception V3
68.16 + 1.12
0.68 + 0.004
0.85 + 0.003
68.38 1.34
0.68 0.004
0.88 0.001
Standard Bottleneck Bottleneck + Attn
61.52 1.20
0.59 = 0.008
Resnet 50V2
61.23 + 1.71
0.59 = 0.008
0.82 = 0.009
61.28 1.54
0.59 + 0.007
0.86 0.004
Standard Bottleneck Bottleneck + Attn.
61.56 + 1.31
0.60 + 0.008
MSR-VE V2E
Resnet 101 V2
61.38 1.24
0.60 0.008
0.83 0.006
61.44 + 1.32
0.60 + 0.009
0.86 - 0.003
Standard Bottleneck Bottleneck + Attn.
61.79 + 1.42
0.60 + 0.012
Inception V3
61,42 1.18
0.60 = 0.013
0.83 0.006
61.68 + 1.23
0.60 - 0.009
0.86 - 0.004
Table 5: Performance of Models
Accuracy (%)
Difference (%)
Dataset
MLP
Linear Classifier
MLB-VD MSR- ASSR-VEE
68.38 61.68
66.94 60.02
-1.44
-166
Table 6: Performance of MLP classifier vs Linear Classifier
17

---

Under review as a conference paper at ICLR 2022
Dataset
Task Accuracy(%)
Task F-1 score
MLB-V2E
75.68 65.23
0.7585 0.6422
MSR-V2E
Table 7: Performance of MLP classifier trained only concepts (without videos)
A. A CLASSIFIER MODEL TRAINED ONLY ON CONCEPTS
The MLP model trained only the concepts (without using video information) achieves higher accuracy (7% for MLB- V2E and 4% for MSR-V2E) than the video classification model. This result indicates that the concepts extracted meaningful for the classification task and there's s still an headroom available if the concept bottleneck network was able to predict these initial concepts better.
A. MORE EXPLANATIONS FROM THE PREDICTION MODEL
Here are few more examples of the prediction by the concept bottleneck model with attention. Figure shows examples from the MLB-V2E dataset and Figure shoows examples from the MSR- V2E dataset. The example vidcos are provided the supplementary materials.
Class: pla
Pieticee Clese. lauf
the sades dae bell
oaaght
agnt rought
Dthers
Concag: Scare
Concest 5c0re
Predicted Cass: Youl
Prudlccad Chees bal
landed
010
9.16
030
325
0.29
Goncept
Figure 8: Examples of the model prediction and their corresponding concepts and their importance scores for MLB-V2E dataset
A.11 MORE DETAILS ON THE DATASETS
Figure [0] shows the number of videos belonging to each category in the MLB-V2E and the MSR- V2E datasets. Though the original MSR- VTT dataset had descriptions of videos, they were general captions and didn't explain any particular class. Since they didn't 't have classification labels and text-based explanations corresponding the labels for the videos, collected the video labels and natural language explanations by crowd-sourcing on Amazon Mechanical Turk. The explanations obtained for these videos and the concepts extracted using CoDEx for both the datasets in the supplementary materials. Since the MSR. ERE dataset is imbalanced, we do some weighted oversampling while training ensure that the models learn to predict all the classes.
provided
18

---

Under review as a conference paper at ICLR 2022
fumily kids
Class vandes
they are singin
mon using car
get
bexrd lodx
driing
Offies
Cthers
0,00
0.05
0.10
0.15
020
0.85
0.90
088
0.40
0.00
0.05
0.10
0.55
0.20
0.25
0.30
Predicted Class cooking
Predicted Class: Music
this video cook ng show
they wetn deaning
person spesking
people dancing sore
they tench how to cut
playing quiser
Others
Others
0000
016
0.33
Cansapt Store
0.15
0120
0.25
13:00
035
0.00
0.55
0.10
Enaragt
0.15
0.20
0.25
0.30
Figure 9: Examples of the model prediction and their corresponding concepts concepts and their importance scores for MSR-V V2E dataset
400
300
COU155
Lampor Straay
sod Setaes
Bussp tupso
Labeet
(6) MLB-VZE Dateset
(b) MSR-VZE Datasel
Figure 10: The number of videos belonging each category on the MLB. MLB-V2E dataset and (b) the MSRRVVE dataset
19


Reviews:
[{"review_id": "tcZoJ05vG-4", "reviewer": "Reviewer_LdBL", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Strengths\n1. The ways the method groups and prunes raw concepts are interesting as these sub-modules are aware of associate labels. \n1. The experimental results are encouraging. Because the concept bottleneck forces to drop much information in the original video, the performance drop seems inevitable; yet the performance degradation is limited. \n\n## Weaknesses\n1. I think the competing approach with this method is the one that uses a set of handcrafted concepts instead of automatically extracted concepts from textual descriptions, while according to the paper, the downside of such handcrafted concepts is that there is no guarantee of having a necessary and sufficient set of concepts. However, this is also the case for textual descriptions, and thus to demonstrate the merit of this method, I think comparison to the approach that uses a set of handcrafted concepts are necessary. One possible comparison is to design such a set of concepts from scratch, and another is to manually extract concepts from textual descriptions. \n1. The necessity of pruning is not clear. According to Figs. 3 and 4, the number of concepts (almost) monotonically increases the performance in senses of both MI and classification. I would say that, perhaps, these less useful concepts do not help a lot, but unless they are harmless (for example, redundant concepts may still increase the chance of correct classification, while if a certain concept is completely useless, the learned FC layer will ignore it), there is no need to remove them. It would be helpful to have some more discussion on the necessity of pruning. \n", "summary_of_the_paper": "This paper presents a method for video classification, which uses textual descriptions of each training video to automatically extract concepts and uses these concepts for classification. For extracting the concepts, the method identifies some phrases in each description. The extracted phrases are grouped based on their semantic similarity and associated labels. Then they are pruned so that only informative phrases are extracted as concepts. For inference, these concepts are identified in a video and based on these identified concepts, an FC layer classifies a video into one of the target classes. The method is evaluated with two datasets MLB-Youtube video activity classification dataset and MSR VTT dataset with extending them with textual descriptions, showing that the concept bottleneck does not impact a lot to the classification performance. The human study demonstrates that identified concepts by their method is better than randomly selected/frequent concepts.", "summary_of_the_review": "I think the paper is well-written, but the merit of the method is not very clear. Since textual descriptions may require a similar burden to handcrafting a set of concepts, the proposed method may just add noises in the set of concepts. Regarding this point, the merit of the method should be experimentally verified (w1). Also, the necessity of pruning is not very clear (w2), which I guess involves the first contribution of the paper. I think the paper will be stronger if these concerns are resolved. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "NhnjMmIHNvj", "reviewer": "Reviewer_9TvQ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n1) CoDEx proposed in this paper provides a more comprehensive understanding of the video feature by extracting concepts from the explanation of the labels and constructing a higher-dimensional concept space. \n2) The pipeline can be easily applied to any supervised video classification architecture.\n\nWeakness: \n1) One of the motivations of the CoDEx is to get concepts without domain experts, but only simple cases are reported in the results. In a simpler domain, the label will be highly correlative to the concepts, so the requirement of experts is low. However, in a more complicated domain, the requirement of experts will become higher. Whether the explanation will still be reasonable is not reported. \n2) This work is more like a \"label decomposition\" based on natural language explanations and has low relevance to video modeling. \n3) The annotation quality of explanations is important since the extracted concepts are from them. However, only two datasets with 5 and 10 classes are reported in this paper. It seems that the annotation cost of this pipeline is high in even normal-size datasets.\n4) In the extraction phase of CoDEx, a fixed set of rules is used to extract \"raw concepts\". The motivation for using sentences as the concept is not clear. Will word-level concepts work? \n\nMinor: Both \"black-box\" and \"blackbox\" are shown in the paper, which is inconsistent. ", "summary_of_the_paper": " This paper proposed an automatic concept discovery and extraction module based on the label explanations and an end-to-end concept bottleneck model for video classification. The result on two constructed datasets (MLB-V2E and MSR-V2E) shows that the architecture can get explainable results without harming the classification performance.\n", "summary_of_the_review": "This paper proposed an end-to-end pipeline to train an explainable video classification model. The paper is targeted to study three interesting research questions, and I particularly liked the discussion section at the end of the paper. However, the experiments lack in some aspects resulting in a less convincing story. \n\n\n######## Post-Rebuttal ########\n\nI appreciate the authors made a good effort to improve their paper based on review comments. However, I still find that the contribution is weak -- experiments are done on only two video datasets with limited classes but given that the method itself is generic. Therefore, I stick with my original rating. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "FDkVpO3tWkM", "reviewer": "Reviewer_uphf", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The authors address the timely problem of interpretable video classification. The paper is well-written, and the authors have also described their method, and data collection process, thoroughly and clearly. And I also appreciate the discussion of future work and weaknesses of the current approach.\n\nHowever, I do feel that the main contribution of the paper -- to generate concepts automatically from natural language descriptions provided by the annotators -- is quite heuristic (combination of NLP post-processing and clustering) and not generalisable across different datasets.\n\nIt is not clear to me how the proposed method can be extended to more common video datasets. For example, on Kinetics or Something-Something, the action classes are fairly \"atomic\", and it is not clear to me what natural language explanations there could for these kind of actions beyond the class label itself. In other words, the \"concepts\" that end up being extracted in this paper, are similar to the class labels themselves in other datasets. And so it is not clear to me how the proposed method can develop interpretable models for such datasets.\n\nI think the paper would also be stronger if the authors compared the annotation effort required to generate the natural language descriptions used in this paper, to the annotation time/cost to collect the original dataset in the first place, compared to the annotation effort required for the manually-defined set of concepts in previous work. Although the proposed concept extraction method is automatic, it still requires significant annotations in the form of natural language descriptions for each video in the dataset. And so the annotation time and effort of this needs to be contextualised.\n\nThe paper would also be more convincing to me if the authors augmented existing, popular video classification datasets (ie Kinetics, Something-Something, Epic Kitchens etc) with descriptions and developed interpretable models for such datasets. Not only are these datasets more common in the community, but I believe that the vocabulary of these datasets are more challenging for the proposed method, compared to for example, the baseball dataset used in this work (since sports videos consist of well-defined actions which are explained by the rules of the sport.)\n\nThe proposed attention mechanism helps to improve the concept prediction accuracy, but not at all on the task classification accuracy. It would also be interesting to see how the fully-connected subnetwork, operating on \"oracle\" concept scores (ie the concept ground truth, rather than the concepts predicted by the network before it) performs. This would show the headroom available if the network was able to predict these initial concepts better.", "summary_of_the_paper": "The authors address the task of concept-based video classification: The resulting model is interpretable, as it first predicts a set of binary \"concepts\", and these concept-features alone are then used for producing the final classification. The main novelty of this work is to propose a method for automatically generating these concepts from natural language descriptions (hence avoiding the need for expert annotations). The authors produce two new datasets of roughly 2000 videos each to validate their method. They also show that their concept-based model performs as well as a standard video classification model on these datasets (ie interpretability does not come at the cost of predictive accuracy). ", "summary_of_the_review": "In summary, the paper is well-written and addresses an important problem of interpretable video classification models. The proposed method does not rely on manual concept labelling, but rather extracts these concepts automatically from natural language descriptions provided by dataset annotators. My concern is that the proposed method does not seem general to me, and I feel that it only works for the specific datasets that the authors have chosen. And although the concept extraction process is automatic, the method requires obtaining natural language descriptions for each video in the dataset, and the annotation cost of this is not discussed at all in the paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "15fd7hhDat0", "reviewer": "Reviewer_zzha", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Here are the pros and cons of the paper:\n\nPros:\n1. This paper presents a quite interesting method which automatically selects the concepts as bottleneck for explanation. This could benefit other interpretability models which have textual descriptions available.\n2. The authors performed a good amount of illustration on the intermediate results after each phase\n3. The proposed datasets could also benefit the related fields/applications\n\nCons:\n1. It seems that there's a lack of ablation studies on each phase and a lack of some decent baseline models for selecting the key concepts. One simple baseline might be, training an MLP together with the information bottleneck model and selecting the ones with high importance. For each phase, it's not quite clear to me how important they are.\n2. It looks like the CoDex also needs some human efforts to tune and intervene the process. Is there any way to empirically prove that they are general enough to process various texts and can obtain similar results with humans?\n3. The authors mentioned that this system is for videos, but it looks like the whole process is quite general, and there is not much correlation with the mentioned issues such as the time span?", "summary_of_the_paper": "This paper works on the interpretability of video understanding problem. With a set of textual descriptions, the authors propose a pipeline called CoDex to extract the key concepts for explaining the classification, in contrast to previous methods which use the predefined classes. The CoDex method contains clearning, extraction, completion, grouping, pruning and vectorization phases to obtain the final concept matrix. The trained final concept bottleneck model can obtain interpretable explanations aligned with human servey, while retaining the performance", "summary_of_the_review": "I think this paper overall contributes an interesting pipeline for automatically extracting relevant concepts for the concept bottleneck model. Several concerns are listed above.\n\n-- after rebuttal --\nThanks for providing the rebuttal and explanations. I've read the authors' responses and other reviews. Overall I still feel the paper is addressing an interesting problem, but agree that the method might limit its applicability. I'll keep my rating (borderline-ish).", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors consider the task of interpretable video classification. First, a set of binary “concepts'' is predicted, and these concept features are then used for classifying a video. The set itself is automatically generated from natural language descriptions, instead of relying on expert annotations. The authors collect two datasets to validate the proposed approach and show that the model can match the performance of a standard video classification model, while being interpretable.

The reviewers felt that the paper was well written and that the method and empirical results were clearly outlined. They also appreciated the empirical results whereby interpretability doesn’t necessarily come at the expense of accuracy and consider interpretability as a desirable property. The main reason for the borderline results is the heuristic nature of the proposed automatic concept labeling and the empirical evaluation against alternative baselines. In particular, one needs to **show that the proposed method generalises to other datasets**. Secondly, one of the main contributions, namely the automatic **concept extraction, still ends up requiring human annotation in the form of narrations**, and this cost should be quantified and contextualised.

I suggest the authors address these points and resubmit.
Decision: Reject