Year: 2021

Abstract:
Recent efforts in interpretable deep learning models have shown that conceptbased explanation methods achieve competitive accuracy with standard end-toend models and enable reasoning and intervention about extracted high-level visual concepts from images, e.g., identifying the wing color and beak length for bird-species classification. However, these concept bottleneck models rely on a domain expert providing a necessary and sufficient set of concepts–which is intractable for complex tasks such as video classification. For complex tasks, the labels and the relationship between visual elements span many frames, e.g., identifying a bird flying or catching prey–necessitating concepts with various levels of abstraction. To this end, we present CODEX, an automatic Concept Discovery and Extraction module that rigorously composes a necessary and sufficient set of concept abstractions for concept-based video classification. CoDEx identifies a rich set of complex concept abstractions from natural language explanations of videos–obviating the need to predefine the amorphous set of concepts. To demonstrate our method’s viability, we construct two new public datasets that combine existing complex video classification datasets with short, crowd-sourced natural language explanations for their labels. Our method elicits inherent complex concept abstractions in natural language to generalize concept-bottleneck methods to complex tasks.

heading: 1 INTRODUCTION
text: Deep neural networks (DNNs) provide unparalleled performance when applied to application domains, including video classification and activity recognition. However, the inherent black-box nature of the DNNs inhibits the ability to explain the output decisions of a model. While opaque decision-making may be sufficient for certain tasks, several critical and sensitive applications force model developers to face a dilemma between selecting the best-performing solution or one that is inherently explainable. For example, in the healthcare domain (Yeung et al. (2019)), a life-or-death diagnosis compels the use of the best performing model, yet accepting an automated prediction without justification is wholly insufficient. Ideally, one could take advantage of the power of deep learning while still providing a sufficient understanding of why a model is making a particular decision, especially if the situation demands trust in a decision that can have severe impacts.
To address the need for model interpretability, researchers have sought to enable model intervention by leveraging concept bottleneck-based explanations. Unlike post hoc explanation methods–where techniques are used to extract an explanation for a given input for an inference by a trained black-box model (Chakraborty et al. (2017); Jeyakumar et al. (2020)), concept bottleneck models are inherently interpretable and take a human reasoning-inspired approach to explaining a model inference based on an underlying set of concepts that define the decisions within an application. Thus far, prior works have focused on concept-based explanation models for image (Kumar et al. (2009); Koh et al. (2020)) and text classification (Murty et al. (2020)). However, the concepts are assumed to be given a priori by a domain expert–a process that may not result in a necessary and sufficient set of concepts. For instance, for bird species identification, an expert may provide two redundant concepts that are possibly correlated, such as wing color and beak color. More critically, prior works have considered simple concepts with the same level of abstraction, e.g., visual elements present in a single image. For more complex tasks such as video activity classification, a label may span multiple frames. Thus, the composing set of concepts will have various levels of abstraction representing relationships of various
visual elements spanning multiple frames, e.g., a bird flapping its wings. Unlike the prior works, we aim to exploit the complex abstractions inherent in natural language explanations to conceptualize such complex events.
Research Questions. In summary, this paper seeks to answer the following research questions:
• How can a machine automatically elicit the inherent complex concepts from natural language to construct a necessary and sufficient set of concepts for video classification tasks?
• Given that a machine can extract such concepts, are they informative and meaningful enough to be detected in videos by DNNs for downstream prediction tasks?
• Are the machine extracted concepts perceived by humans as good explanations for the correct classifications?
Approach. This paper introduces an automatic concept extraction module for concept bottleneckbased video classification. The bottleneck architecture equips a standard video classification model with an intermediate concept prediction layer that identifies concepts spanning multiple video frames. To compose the concepts that will be predicted by the model, we propose a natural language processing (NLP) based automatic Concept Discovery and Extraction module, CODEX, to extract a rich set of concepts from natural language explanations of a video classification. NLP tools are leveraged to elicit inherent complex concept abstractions in natural language. CODEX identifies and groups short textual fragments relating to events, thereby capturing the complex concepts from videos. Thus, we amortize the effort required to define and label the necessary and sufficient set of concepts. Moreover, we employ an attention mechanism to highlight and quantify which concepts are most important for a given decision.
To demonstrate the efficacy of our approach, we construct two new datasets–MLB V2E (Video to Explanations) for baseball activity classification and MSR-V2E for video category classification– that combine complex video classification datasets with short, crowd-sourced natural language explanations for their corresponding labels. We first compare our model against the existing standard end-to-end deep-learning methods for video classification and show that our architecture provides additional benefits of an inherently interpretable model with a marginal impact on performance (less than 0.3% accuracy loss on classification tasks). A subsequent user study showed that the extracted concepts were perceived by humans as good explanations for the classification on both the MLB-V2E and MSR-V2E datasets.
Contributions. We summarize our contributions as follows.
• We propose CoDEx, a concept discovery and extraction module that leverages NLP techniques to automatically extract complex concept abstractions from crowd-sourced, natural language explanations for a given video and label–obviating the need to manually define a necessary and sufficient set of concepts.
• We evaluate our approach on complex video classification datasets and show that our model attains high concept accuracies while maintaining competitive task performance with standard end-to-end video classification models.
• We also augment the concept-based explanation architecture to include an attention mechanism that highlights the importance of each concept for a given decision. We show that users prefer our concept extraction method over baseline methods to explain a given label.
• We construct two new public datasets, MLB-V2E and MSR-V2E, that combine complex video classification datasets with short, crowd-sourced natural language explanations and labels.

heading: 2 RELATED WORK
text: There is a wide array of works in explainable deep learning for various applications. This work focuses on the concepts-based explanations for video classification, and this section provides an overview of the existing literature for overlapping domains. Concept-Based Explanations for Images and Text. A number of existing works consider conceptbottleneck architectures where models are trained to interact with high-level concepts. Generally, the
approaches are multi-task architectures, where the model first identifies a human-understandable set of concepts and then reasons about the identified concepts. Until now, the applications have been limited to static image and text applications. Koh et al. (2020) used pre-labeled concepts provided by the dataset to train a model that predicts the concepts, which is then used to predict the final classification. However, the caveat is that the concepts had to be manually provided. Ghorbani et al. (2019) and Yeh et al. (2020) proposed approaches that automatically extract groups of pixels from the input image that represent meaningful concepts for the prediction. They were designed largely for image classification and extract concepts directly from the dataset. Kim et al. (2018) propose a post-hoc explanation method that returns the importance of user-defined concepts for a classification. In the mentioned works, the concepts have been limited to simple concepts and are not suited for complex tasks such as video classification where we have complex concepts that may span multiple frames with various levels of abstraction. Explanations for Video Classification Other approaches have been considered to explain video classification and activity recognition. Chattopadhay et al. (2018) applied GradCAM and GradCAM++ to video classification, where for each frame, the important region of the frame to the model is highlighted as a heatmap. Hiley et al. (2020) extract both spatial and temporal explanations from input videos by highlighting the relevant pixels. However, these are post-hoc techniques that focus on explaining black-box models, whereas our approach enables concept-bottleneck methods for video classification that are intended to be inherently interpretable and intervenable. Video Captioning. In recent years, there is a large number of works (Pan et al. (2017); Gao et al. (2017); Wang et al. (2018); Yan et al. (2019); Zhou et al. (2018); Chen & Jiang (2021); Yu et al. (2017)) on video captioning. While they also employ natural language techniques, these works are tangential to generating text explanations for classifications, since they are merely describing the video. Our model provides an explanation justifying the classification of the video. Similarly, the associated datasets such as MSR-VTT (Xu et al. (2016)) only have videos with ground truth captions that only describe the video without the context of a classification–which often results in concepts that do not pertain to a classification. Semantic Concept Video Classification. The closest works to this paper is the body of work in semantic concept video classification (Fan et al. (2004; 2007)), where the concepts are defined as salient objects that are visually distinguishable video components. The concepts in these works are simple objects detected in the videos and are not complex enough to capture the semantics of events that happen over multiple frames of the videos. These works typically used traditional SVM-based video classifiers. Assari et al. (2014) represent a video category based on the co-occurrences of the semantic concepts and classify based on the co-occurrences, but their method requires a predefined set of concepts. Thus, we now present the methodology behind our automatic concept extraction for concept bottleneck video classification.

heading: 3 CONCEPT DISCOVERY AND BOTTLENECK VIDEO CLASSIFICATION
text: This work introduces CoDEx, an automatic concept extraction method from natural language explanations for concept-based video classification. Figure 1 depicts the overall concept-bottleneck pipeline, composed of CoDEx and the concept bottleneck model. CoDEx extracts a set of concepts
from natural language explanations that will comprise the bottleneck layer for the video classification model. We first formalize the overall problem and then provide the methodology for both modules.
Problem Formalization. We assume that we have a training dataset {(xn, ln)}Nn=1 = D of videos xn with a label ln ∈ L, where L is a predefined set of possible class labels for the video. Each video is represented as a sequence of frames f ∈ F where F is the set of video frames. Thus video xn = 〈fn0, fn1, . . . , fnT 〉 , where fnt represents frame t of video n. For each video xn, we form a label-explanation pair (ln, en), where en is a (short) natural language document explaining the given label ln. If multiple annotators contribute to an explanation for video-label pair, (xn, ln) , then these are concatenated to form en. The full set of pairs E = {(ln, en)}Nn=1 is the explanation corpus. Thus, the design goals are:
• Concept Discovery and Extraction (CODEX) Module: Given the explanation corpus, first produce an N × K concept matrix, C, where the (n, k)th element is 1 if the nth explanation contains discovered concept k and 0 otherwise. We call the nth row cn, the concept vector for video xn. K is the total number of discovered concepts.
• Concept Bottleneck Model: Given a concept matrix, C, the second goal is to train a concept bottleneck model such that for a given video xi, we predict a concept vector ci– which indicates the presence or absence of concepts and their importance scores. The model then makes use of ci to make the final video classification.
label, ln explanation, en
strike The batter did not swing. The ball was in the strike zone.
foul The batter hit the ball into the stands and it landed in foul territory.
ball The hitter didn’t swing. The ball was outside the strike zone.
none The video did not load.
out the batter hit the ball and it was caught by the fielder
label, ln concepts, cn
strike {the batter did not swing, the ball was in the strike zone}
foul {the batter hit the ball, it landed in foul territory}
ball {the batter did not swing, the ball was outside the strike zone}
out {the batter hit the ball, it was caught by the fielder}
Grouped conceptsCompleted concepts Text Pruned conceptsText Removed text
Explanation Corpus Ɛ Extracted Concepts
Figure 2: Running example for all six stages of the discovery pipeline module. The left table is the explanation corpus, with highlighted fragments to be modified. The right table contains the discovered concepts. The detailed step-by-step modifications are provided in Appendix A.1.

heading: 3.1 CODEX: CONCEPT DISCOVERY AND EXTRACTION MODULE
text: We now describe CoDEx, that extracts concepts from the explanation corpus, E . The automatic extraction of the significant concepts is done in 6 steps, as outlined in Fig. 1. These are: cleaning, extraction, grouping, completion, pruning, and vectorization, which produce the final concept matrix, C. Each of these steps are described below and illustrated with an example corpus depicted in Figure 2.
Cleaning. We remove explanations associated with corrupted or unlabeled videos from the explanation corpus. In Figure 2, this phase would remove the fourth row with the "none" label.
Extraction. The objective of this phase is to identify sentence constituents relevant to explaining the label. These text fragments, short sequences of words that appear in the document, are referred to as raw concepts. To achieve this, the cleaned explanation corpus is tokenized then passed through a pretrained constituency parser to recursively decompose the sentences. At each level of the constituency hierarchy, the text fragments are evaluated to determine whether they constitute a candidate raw concept. The rules for candidate raw concepts include the inclusion and exclusion rules below and follow the widely adopted Universal POS tag naming convention for token types (Petrov et al. (2012)). Every constituency parsed phrase that satisfies one of the two inclusion rules and not the exclusion rule is considered a candidate concept.
After the extraction process is completed, we have a set of raw concepts, K̃, and each video is associated with a subset of these raw concepts. An example of extracted raw concepts, K̃, can be found in Appendix A.1.
Completion. There are instances where the pretrained constituency parser will split sentences midway through a text fragment in one sentence that was kept whole in another. For instance, in Figure 2, the constituency parser splits the explanation for "foul" such that "the batter hit the ball" is incorrectly excluded from the raw concepts. To ensure that those concepts are captured, we perform a substring lookup of each raw concept through all documents of the explanation corpus and count an explanation as containing a raw concept if it contains the corresponding raw concept as a substring. This does not change the number of raw concepts identified but increases their frequency counts.
Grouping (similar raw concepts). When identical text fragments are identified in different explanations, they are counted directly as the same raw concept. However, we would ideally like to treat superficially different concepts as the same if they essentially carry the same meaning, e.g., Figure 2 highlights two different raw concepts that carry the same meaning and hence can be grouped. For this, we use agglomerative clustering (Müllner (2011)) approach that measures the degree of difference between pairs of raw concepts and groups them together if they are similar enough. Our key contribution here is the distance metric used in clustering which is a novel measure of meta-distance between raw concepts. This measures the difference between concepts based on two aspects of the raw concepts: their linguistic difference and their difference in terms of the label categories with which they are associated.
We define meta-metric, d, as combining a linguistic distance, dtext (capturing linguistic difference) as well as a meta-metric, dlabel (capturing the difference in the labels associated with each raw concept). More formally, for two raw concepts κi, κj ∈ K̃ our distance is linear combination:
d(κi, κj) = dtext(vi,vj) + λdlabel(ni,nj) (1)
where vi is a sentence embedding for the text fragment of concept κi (e.g., based on the BERT model Devlin et al. (2019)), dtext is a standard distance measure between vectors (e.g., cosine distance), dlabel is a meta-distance which aims to capture the similarity between two label count vectors, and λ is a hyperparameter controlling the relative importance between textual and label distance. The inclusion of a label distance helps to distinguish between concepts that are superficially linguistically very simlar, but have very distinct meanings within the domain of interest. For instance, without the dlabel, the concepts “the ball passed inside the strike zone” and “the ball passed outside the strike zone” will be grouped together though they are very different concepts as they have a very small dtext. We provide a more formal definition of the meta-metric dlabel more formally and provide some intuition behind its construction in Appendix A.3. We also exclude concept groups which occur very rarely in the explanation corpus, with frequency less than some small threshold, t.
Pruning. Here, we seek a compact subset of concepts that, together, capture a high degree of information about the label while maintaining interpretability. More formally, after grouping, we have a set of raw concepts K̃ = {κ1, . . . , κJ}, and we seek some subset of maximally informative concepts K? = {κj1 . . . , κjK} ⊆ K̃. To see what is meant by maximally informative, consider a randomly selected entry in the explanation corpus (l, e). We define a binary random variable, Cj for each raw concept κj , and for any concept set K = {κj1 , . . . , κjK}, random vector CK = [C1, . . . ,CK ], such that Cj = 1 if κj ∈ e and 0 otherwise. Y is the random variable which takes label l. We wish to choose the smallest subset of concepts such that the mutual information (MI)1 between chosen concepts, K, and label, Y , given by I(Y ;CK), is greater than a threshold fraction, γ < 1 of the MI between the label and the complete concept vector, I(Y ;CK̃). That is to say we wish to find K which satisfies:
I(Y ;CK) ≥ γI(Y ;CK̃) (2)
and where there is no subset K′ ⊆ K̃ with |K′| < |K| which also satisfies Equation equation 2. In practice, this is infeasible as the problem is combinatorial. However, we note that f(K) = I(Y ;CK) is a monotone submodular set function of K̃. Given this, if we recursively construct a set of size K, by greedily adding single concepts that most improve the MI, the resulting set will be at least 1− 1e
1We use the standard definition of mutual information (MI) for discrete random variables (MacKay (2003)).
as good as the most informative set of size K (Nemhauser et al. (1978)). Therefore, we guarantee a highly-informative set K? by iteratively adding concepts to those previously selected, greedy with respect to the MI, until we have a set that satisfies Equation 2.
Vectorization. Each concept κjk ∈ K? is given a unique index k ∈ {1, . . . ,K}, and each datapoint, xn is associated with a concept vector cn = (cn1, . . . , cnK), where cnk = 1 if κjk ∈ en and 0 otherwise, indicating the presence or absence of the kth concept in the nth explanation. The collection of all the concept vectors gives an N ×K concept matrix, C.

heading: 3.2 CONCEPT BOTTLENECK MODEL
text: We use the videos, the extracted concepts from CoDEx, and the labels to train an interpretable concept-bottleneck model to predict the activity and the corresponding concepts. Figure 1 shows the overview of our bottleneck architecture. The activity label, the concepts, and the corresponding concept scores are the outputs of the interpretable model and are indicated by dotted arrows in Figure 1.
Our bottleneck model architecture is based on the standard end-to-end video classification models where we use convolutional neural network-based feature extractors pretrained on the Imagenet dataset Deng et al. (2009) to extract the spatial features from the videos. The features are then passed through temporal layers that can capture features across multiple frames which in turn is bottle-necked to predict the concepts. Lastly, we deploy an additive attention module (Bahdanau et al. (2014) that gives the concept score αc indicating the importance of every concept to the classification. The attention module also improves the interpretability of the the bottleneck model by indicating the key concepts for classification and this is evaluated in section 5. More details regarding the model architecture and hyper-parameters are in the Appendix A.5
Model loss function. The entire bottleneck classification model is trained in an end-to-end manner. Since the concepts are represented as binary vectors, we use sigmoid activation on the concept bottleneck layer and binary categorical loss function as the concept loss. The final layer of the classifier has softmax activations and categorical cross-entropy as the classification loss function. Thus, the overall loss function of the model is the sum of concept loss and the classification loss. The hyperparameter β controls the tradeoff between concept loss, LC , versus classification loss, LY as shown in equation 3. The full expansion of the equation is located in Appendix A.5.
Loss(L) = 1
N N∑ n=1 (LYn + β × LCn) where β > 0 (3)
Testing phase. Given an input test video, the model provides us with the activity prediction (label of the video), a concept vector indicating the relevant concepts that induced this classification and the concept importance score for each concept. By retrieving the phrase representing the concepts present in the video, the result obtained is a human-understandable explanation of the classification.

heading: 4 IMPLEMENTATION
text: To demonstrate our automatic concept extraction method, we construct two new datasets - MLB-V2E (Video to Explanations) and MSR-V2E, which combines short video clips with crowd-sourced classification labels and corresponding natural language explanations. For both datasets, we obtained a video activity label and natural language explanations for that label by crowd-sourcing on Amazon Mechanical Turk and used unrestricted text explanations to extract concepts automatically. For IRB exemption and compensation information, please refer to the Ethics Statement. MLB-V2E Dataset: We used a subset of the MLB-Youtube video activity classification dataset introduced by Piergiovanni & Ryoo (2018)–which had segmented video clips containing the five primary activities in baseball: strike, ball, foul, out, in play. We preprocessed the dataset and extracted 2000 segmented video clips where each video was 12 seconds long, 224×224 in resolution, and recorded at 30 fps. To ensure that the quality of explanations is good, we screened over 450 participants. Based on their baseball knowledge, 150 participants were qualified to provide the natural language text explanations for our video clips. We have included a sample of our screening survey, the primary survey, and the explanations collected in the supplementary materials.
MSR-V2E Dataset: For this dataset, we used 2020 video clips from the MSR VTT dataset introduced by Xu et al. (2016). The MSR-VTT dataset has general videos from everyday life and descriptions of these videos associated with them. Each video clip is between 10-30 seconds long, and approximately 200 participants provided the labels and explanations to construct the MSR-V2E dataset. The videos are classified into ten categories: Automobiles, Cooking, Beauty and Fashion, News, Science and Technology, Eating, Playing Sports, Music, Animals, and Family (more details in Appendix A.11). Training: All our models were trained on 2 × Titan GTX GPUs using Adam optimizer. A summary of our entire model architecture and a trained model is provided in the supplementary materials.

heading: 5 RESULTS
text: Number of extracted concepts. Table 2 shows that the system extracted 80 concepts and 62 concepts from the explanation corpus of MLB-V2E and MSR-V2E respectively. The number of concepts remaining after the pruning phase is determined by the cumulative Mutual Information(MI) threshold. To identify the best threshold, we plotted the number of concepts at different thresholds versus performance of the model as shown in Figure 3. We found that the task classification performance did not increase after a certain number of concepts and that optimal spot for the number of concepts corresponded to 90% Mutual Information (Appendix A.6).
Comparing concept-bottleneck models to baselines. We adopt model architectures and hyperparameters from standard well-performing approaches that fall under 3 categories: 1) without concept bottleneck, 2) with concept bottleneck, 3) with concept bottleneck and attention. We compared the performance of models with the bottleneck layer with standard video classification models without a concept bottleneck layer. We find that, though the latent space was constrained to the limited set of concepts extracted from the explanation corpus, concept models performed as well as the uncon-

heading: Dataset Concepts Extraction Task Accuracy(%) Task F1-score Concept AUC
text: strained models, on both datasets. We also find that the addition of the attention layer improves the concept prediction of the models. Table 3 shows that concept bottleneck models achieved comparable task accuracy to standard black-box models on both tasks, despite the bottleneck constraint while achieving high concept prediction performance. Appendix A.7 shows the performance with other feature extractors.
Ablation Study of CoDEx. We performed an ablation study to highlight the impact of CODEX’s components. For sentence-level concept extraction, we replaced the extraction phase with word-level concepts extracted from the explanation corpus. We also evaluated how removing the grouping and pruning phases would impact performance. Table 4 shows the results of our study. We find that using word-level concepts significantly reduced the performance of predicting the task and the concepts. The grouping and pruning stages had a greater impact on the concept prediction performance that affected the explainability of the model.
Concept scores for interpretability. Not only does the attention module increase performance in concept prediction, but it also improves the explainability of the bottleneck model by providing an importance score for the concepts. Figure 4 shows the explanation from the concept bottleneck model with attention on a test sample from the two datasets. More examples can be found in Appendix A.10. The title shows the classification label, the y-axis indicates the top-3 concepts predicted as present in the video clip, and the x-axis corresponds to the concept score. Others refers to the sum of the importance of all the remaining concepts.
Human study to evaluate concepts’ explainability. We performed a Mechanical Turk study to evaluate the explainability of our extracted complex concepts to the end-users. The participants were asked to select from four different options (presented in random) of what they consider to be the best possible Explanation for the classification of a given video. The four options are: Complex concepts predicted models without attention, Complex concepts predicted by models with attention, Concepts of a random video not belonging to the same predicted class and a Random set of 2-5 concepts from the set of the most frequent concepts. The methodology of this study was inspired by Chang et al. (2009)’s paper. Figure 5 presents the aggregated results of the Mechanical Turk study. The complex concepts predicted by the concept bottleneck model with attention was considered as the preferred explanation by 68% and 57% of the responses in the MLB-V2E and MSR-V2E datasets respectively followed by the concepts bottleneck models without attention in 20% and 28% of the responses for
the two datasets. The presented confidence intervals are calculated using the bootstrap method as described by DiCiccio & Efron (1996) for 95% confidence.

heading: 6 DISCUSSION
text: Annotation effort. Although CODEX requires collecting a large explanation corpus, prior work requires two studies: one study to identify the set of essential concepts, and a second study to annotate the videos with the essential concepts–whereas CODEX only requires a single study. Further, if the vocabulary of concepts is large, the annotation process would be inconvenient for the user. Moreover, natural language explanations can express richer compositions of concepts rather than simply identifying the presence or absence of an individual concept. Thus, CODEX’s annotation efforts are more expressive, less expensive and less cumbersome than prior works. Representative Concept. Our concept extraction method selects the most frequent concept in a grouped cluster as the representative concept. In general, the most frequent concept suffices to explain a particular component of the complex activity. However, there were some instances where the most frequent concept would have a specific terminology rather than a general term, e.g., "left fielder" is a subclass of "outfielder." Future work can strive towards generating the representative concept for a cluster, as opposed to opting for the most frequent or popular phrasing. Preserving Spatial-temporal Semantics. Our model’s output explanation currently provides a set of activated concepts along with their score. However, they do not capture the spatial and temporal relationships between concepts. Some rich concepts implicitly embed spatial and temporal properties, e.g., “the batter hit the ball on the ground" implies the following sequence: a batter swung at a ball, made contact with the ball, and the ball landed on the ground. However, if the generated set of concepts is limited to less informative concepts, e.g., “the batter," the spatial and temporal ordering of concepts matters. Future work can generalize the architecture to generate concept-based natural language explanations that explicitly preserve spatial-temporal semantics. Neural-symbolic Reasoning. Our model’s reasoning layers are inherently black-box in nature, i.e., the concept vectors are fed into a fully connected network. To further bolster human-machine teaming and interpretability, the final classification model can be replaced with a rule-based model–analogous to prior works that fuse deep learning inferences with symbolic reasoning layers for complex event detection (Xing et al. (2020); Vilamala et al. (2019)).

heading: 7 CONCLUSION
text: The remarkable performance of deep neural networks is only limited by the stark limitation in clearly explaining their inner workings. While researchers have introduced feature highlighting explanation techniques to provide insight into these black-box models, concept-bottleneck models offer a promising new approach to explanation by decomposing application tasks into a set of underlying concepts. We build upon concept-based explanations by introducing an automatic concept extraction module, CoDEx, to a general concept bottleneck architecture for identifying, training, and explaining video classification tasks. In coalescing concept definitions across crowd-sourced explanations, CoDEx amortizes the expertise of concept definition while removing the burden from the model developer. We also show that our method provides reasonable explanations for classification without compromising performance compared to standard end-to-end video classification models.

heading: ETHICS STATEMENT
text: IRB Exemption and Compensation. This research study has been certified as exempt from review by the IRB and the participants were compensated at a rate of 15 USD per hour for a total of 920.36 USD spent.
Dataset privacy. There was no personally identifiable information collected at anytime during the turk study. The responses provided by the mechanical turkers that are present in the dataset are completely anonymous.

heading: REPRODUCIBILITY STATEMENT
text: The entire code with detailed comments are provided in the supplementary materials. The model architectures and hyper-parameters used are discussed in Appendix A.5. All the plots and graphs can be obtained by running the code without modifications.

heading: A APPENDIX

heading: A.1 RUNNING EXAMPLE TO DESCRIBE THE CONCEPT DISCOVERY PIPELINE
text: Example (A simple Explanation corpus). Consider using baseball domain with a few meaningful labels, a null label L = {strike, ball, foul, out, none} and five entries in the explanation corpus E:
id, n label, ln explanation, en 1 strike The batter did not swing. The ball was in the strike zone. 2 foul the batter hit the ball into the stands and it landed in foul territory 3 ball The hitter didn’t swing. The ball was outside the strike zone. 4 none The video did not load. 5 out the batter hit the ball and it was caught by the fielder
Example (After Cleaning Phase). The none label entry is removed after the Cleaning phase
id, n label, ln explanation, en 1 strike The batter did not swing. The ball was in the strike zone. 2 foul the batter hit the ball into the stands and it landed in foul territory. 3 ball The hitter didn’t swing. The ball was outside the strike zone. 4 out the batter hit the ball and it was caught by the fielder
Example (After Extraction Phase). The raw concepts are extracted based on the defined rules discussed in Table 1 corresponding to each explanation with the help of a pre-trained constituency parser.
id, n label, ln raw concepts, K̃ 1 strike The batter did not swing, The ball was in the strike zone 2 foul the ball into the stands, it landed in foul territory 3 ball The hitter didn’t swing, The ball was outside the strike zone 4 out the batter hit the ball, it was caught by the fielder
Example (After Completion Phase). The concept ’the batter hit the ball’ was not extracted by the Extraction phase for the Id 2 explanation in this corpus. This missing concept is retrieved through the sub-string matching.
id, n label, ln raw concepts, K̃ 1 strike The batter did not swing, The ball was in the strike zone 2 foul the batter hit the ball, the ball into the stands, it landed in foul territory 3 ball The hitter didn’t swing, The ball was outside the strike zone 4 out the batter hit the ball, it was caught by the fielder
Example (After Grouping Phase). We show the grouped concepts for this example corpus Concept-index, i Concept groups
1 [The batter did not swing, The hitter didn’t swing] 2 [the batter hit the ball, the batter hit the ball] 3 [The ball was in the strike zone] 4 [The ball into the stands] 5 [it landed in foul territory] 6 [The ball was outside the strike zone] 7 [it was caught by the fielder]
Example (After Pruning Phase). The concept ’the ball into the stands’ of Concept-Index 4 from previous table was not contributing much and hence was pruned.
Concept-index, i Concept groups 1 [The batter did not swing, The hitter didn’t swing] 2 [the batter hit the ball, the batter hit the ball] 3 [The ball was in the strike zone] 4 [it landed in foul territory] 5 [The ball was outside the strike zone] 6 [it was caught by the fielder]
Example (After Vectorization). After pruning, each sample is mapped to their corresponding concept vector. The value of concept vector at index i is 1 if en has the concept with index i, else, it’s 0. The Matrix containing all the concept vectors is called the Concept Matric, C
id, n label, ln Concept Vector, cn 1 strike [1,0,1,0,0,0] 2 foul [0,1,0,1,0,0] 3 ball [1,0,0,0,1,0] 4 out [0,1,0,0,0,1]

heading: A.2 EXAMPLE CONSTITUENCY TREE
text: The explanation n = 1 from Example A.1 is decomposed into the constituency tree shown in Figure 6. The parser gives a hierarchy of constituents. Our method traverses through this tree and selects the constituents that satisfy the rules discussed in Table 1. It is important to note that rules can be added or deleted at this step based on the requirements.

heading: A.3 META-DISTANCE FOR LABEL BASED PROXIMITY
text: At the end of the Completion Phase we define a count for each raw concept, κi ∈ K̃, given by Mi. And for each label category l ∈ L, we define a label count for raw concept κi as mil where i is the index of the concept. These count the presence of raw concepts explanations, and∑
l∈Lmil =Mi. Finally we group together raw concept κi’s label counts into a label count vector mi = [mi1, . . . ,mi|L|]
Now we describe the meta-metric dlabel used in the Grouping phase more formally and provide some intuition behind its construction. Consider that we have two raw concepts κi, κj ∈ K̃ and label count vectors mi and mj . We next assume that vector mi constitutes Mi i.i.d. draws from a categorical distribution with unknown parameters µi = (µil) |L| l=1, where µil is the probability that a randomly selected occurrence of raw concept i belongs to an entry in the explanation corpus with label category l. Our label distance dlabel is the evidence ratio between the count vectors, mi and mj , being drawn from independent categorical distributions (modelMindp) versus them being drawn from the same distribution (modelMcomb). More precisely,
dlabel(mi,mj) = p(mi,mj |Mindp) p(mi,mj |Mcomb)
Note that this is not a true distance between count vectors as two identical count vectors do not have a distance of zero. Nonetheless, it satisfies the other requirements of a metric: non-negativity, symmetry and the triangle inequality, and two vectors that are more (less) likely to come from the same multinomial will have a distance less (more) than 1.
To evaluate the label distance we must calculate the evidence for various categorical samples given the model p(m|µ,M) . For simplicity, we assume total count M is known and define a Dirichlet prior p(µ|α1) where 1 is the vector of all 1s (this makes the simplifying assumption that the prior is symmetric). The evidence for m is then:
p(m|α) = ∫ p(m|µ,M)p(µ|α1)dµ
The label meta-metric is the evidence ratio given by:
dlabel(mi,mj) = p(mi|α)p(mj |α) p(mi +mj |α)
For computational efficiency (and since it did not appear to affect results measurably) we use an approximation for p(m|α) in our calculation of dl. We first evaluate the expected parameter of the posterior distribution given count vector m, namely
µ̃ = E[µ|α,m]
then evaluate the evidence for m conditioned on µ̃, i.e.
p(m|µ̃) = K∏ k=1 ( mil + α Mi +Kα )mil We then calculate log dlabel(mi,mj) and exponentiate to improve precision. After grouping, the raw concept within a cluster with highest frequency is identified as the representative concept of that cluster.

heading: A.4 LANGUAGE MODELS

heading: A.4.1 CONCEPT EXTRACTION
text: After obtaining the free form textual explanations for both, we first cleaned them by removing explanations associated with corrupted video files and the videos which were labelled incorrectly. We then considered three different Spacy’s pretrained constituency parsers: en_core_web_lg, en_core_web_md, en_core_web_sm to parse the explanations and extract raw concepts based on the rules discussed in section 3.1. We found that, the parser en_core_web_lg was more accurate in identifying the constituents and resulted in better concept extraction.

heading: A.4.2 CONCEPT GROUPING
text: For text distance we embedded the raw concepts with a sentence encoder and we experimented with two models: paraphrase-distilroberta-base-v1 (distil) Sanh et al. (2019) and stsb-roberta-base (stsb) Reimers & Gurevych (2019) both using the sentence_transformer python library, and evaluated a variety of distance metrics within the resulting 768 dimensional space, including: Chebyshev (infinity norm), manhattan, Euclidean and cosine distances.
And to cluster the semantically similar concepts together using agglomerative clustering Müllner (2011), we evaluated a variety of distance metrics within the resulting 768-dimensional space, including: Our proposed meta-distance metric, Chebyshev (infinity norm), manhattan, Euclidean and cosine distances. To select hyperparameters, including: choice of sentence embedding model, distance metric for sentence embeddings, prior α for label distance, and relative importance factor λ we performed a grid search and selected the values that resulted in well-formed clusters. Based on our experiments (Provided in supplementary materials), we found that stsb encoder with our proposed meta-distance metric resulted in the best grouping of concepts. Note: dtext can be either cosine or manhattan distance as they gave similar clusters.
After clustering, we set the frequency occurrence threshold of 3 and removed the rare concept groups which occurred less than this threshold. Then, pruning was done using 90% of mutual information score as discussed in section 3.1 which resulted in 80 significant concepts for our MLB-V2E dataset and 62 concepts for MSR-V2E dataset as shown in Table 2 and Figure 7. Therefore, each video was associated with a binary concept vector of shape [1×k] where k is the number of concepts, indicating the presence and absence of each concept.

heading: A.5 THE CLASSIFICATION MODELS
text: We considered three different feature extractors : Resnet 50v2 He et al. (2016), Resnet 101v2 He et al. (2016) and InceptionV3 Szegedy et al. (2016) models and pretrained on the Imagenet dataset to extract features from each frame of our video clips. We excluded the final classification layer from these models and did a global maxpool across the width and height such that we get a 2048 size feature vector for every frame. We then concatenate the features together, resulting in a [2048× 360] feature matrix for every video where 360 is the number of frames per video.
For the Temporal Layer, we considered both temporal convolution Lea et al. (2017) and LSTM Hochreiter & Schmidhuber (1997) based architectures which are good at extracting temporal features and found that Temporal CNNs outperformed LSTM by a significant amount. And the Bottleneck Layer is a dense layer with k neurons and hence the output is a vector of shape [1 × k] where k is the number of significant concepts. We introduced an attention layer in the concept-bottleneck model that gives the concept score for each concept. The final fully connected layer is implemented with L neurons (L classes) which predicts the class from the video.
Model Loss function Assuming s is the feature vectors obtained from the Feature extractors
Loss(L) = 1
N N∑ n=1 (LYn + β × LCn) (4)
= 1
n n∑ i=1 β l∑ k=1 [ −cik log(fσ(sk))− (1− ck log(1− fσ(sk))) ] − m∑ j=1 yj log fS(sj)  where fσ(si) = 1
1 + e−si and fS(si) = esi∑m j=1 e sj and β > 0

heading: A.6 SELECTING CONCEPTS BASED ON THE MUTUAL INFORMATION(MI)
text: Figure 7 shows the plot between cumulative MI and the number of concepts after pruning. As discussed in Section 5 the sweet spot for the number of concepts corresponded to 90% of the cumulative MI beyond which there was no gain in classification performance as we increased the number of concepts.

heading: A.7 PERFORMANCE OF MODELS
text: Table 5 shows the performance of all the models with different feature extractors. Each model was trained thrice and the mean and standard deviations are reported. We find that models with Inception V3 as the feature extractor performed the best. Adding attention mechanism greatly improved the performance of concepts prediction and also achieved higher accuracies than the concept bottleneck models without attention.

heading: A.8 THE RELATIONSHIP BETWEEN CONCEPTS AND THE CLASSIFICATION TASK
text: To understand the relationship between the extracted the concepts and the task classification, we compared the performance of the Concept Bottleneck models with a) MLP classifier b)Linear Classifier as the final classification layers. The results showed that there was approximately 2% drop in classification performance when using a Linear Classifier instead of an MLP. This indicates that, the classification task is not a simple linear combination of the extracted concepts and the composition of concepts is important.

heading: A.9 A CLASSIFIER MODEL TRAINED ONLY ON CONCEPTS
text: The MLP model trained on only the concepts (without using video information) achieves a higher accuracy (7% for MLB-V2E and 4% for MSR-V2E) than the video classification model. This result indicates that the concepts extracted are meaningful for the classification task and there’s still an headroom available if the concept bottleneck network was able to predict these initial concepts better.

heading: A.10 MORE EXPLANATIONS FROM THE PREDICTION MODEL
text: Here are a few more examples of the prediction by the concept bottleneck model with attention. Figure 8 shows examples from the MLB-V2E dataset and Figure 9 shows examples from the MSRV2E dataset. The example videos are provided in the supplementary materials.

heading: A.11 MORE DETAILS ON THE DATASETS
text: Figure 10 shows the number of videos belonging to each category in the MLB-V2E and the MSRV2E datasets. Though the original MSR-VTT dataset had descriptions of videos, they were general captions and didn’t explain any particular class. Since they didn’t have classification labels and text-based explanations corresponding to the labels for the videos, we collected the video labels and natural language explanations by crowd-sourcing on Amazon Mechanical Turk. The explanations obtained for these videos and the concepts extracted using CoDEx for both the datasets are provided in the supplementary materials. Since the MSR-V2E dataset is imbalanced, we do some weighted oversampling while training to ensure that the models learn to predict all the classes.

Review summaries:
This paper works on the interpretability of video understanding problem. With a set of textual descriptions, the authors propose a pipeline called CoDex to extract the key concepts for explaining the classification, in contrast to previous methods which use the predefined classes. The CoDex method contains clearning, extraction, completion, grouping, pruning and vectorization phases to obtain the final concept matrix. The trained final concept bottleneck model can obtain interpretable explanations aligned with human servey, while retaining the performance

The authors address the task of concept-based video classification: The resulting model is interpretable, as it first predicts a set of binary "concepts", and these concept-features alone are then used for producing the final classification. The main novelty of this work is to propose a method for automatically generating these concepts from natural language descriptions (hence avoiding the need for expert annotations). The authors produce two new datasets of roughly 2000 videos each to validate their method. They also show that their concept-based model performs as well as a standard video classification model on these datasets (ie interpretability does not come at the cost of predictive accuracy). 

This paper presents a method for video classification, which uses textual descriptions of each training video to automatically extract concepts and uses these concepts for classification. For extracting the concepts, the method identifies some phrases in each description. The extracted phrases are grouped based on their semantic similarity and associated labels. Then they are pruned so that only informative phrases are extracted as concepts. For inference, these concepts are identified in a video and based on these identified concepts, an FC layer classifies a video into one of the target classes. The method is evaluated with two datasets MLB-Youtube video activity classification dataset and MSR VTT dataset with extending them with textual descriptions, showing that the concept bottleneck does not impact a lot to the classification performance. The human study demonstrates that identified concepts by their method is better than randomly selected/frequent concepts.