Disentangled Concepts Speak Louder Than Words: Explainable Video Action Recognition
Authors: Jongseo Lee
Wooil Lee
Gyeong-Moon Park
Seong Tae Kim
Jinwoo Choi
Venue: NeurIPS 2025 spotlight ()
Keywords: Video Action Recognition
Explainable AI
XAI
Concept
Disentangling
Motion Dynamics
Context
Primary area: applications
TL;DR: We propose DANCE, a novel explainable video action recognition framework that provides clear and structured explanations by disentangling motion dynamics and spatial context concepts.

Abstract: Effective explanations of video action recognition models should disentangle how movements unfold over time from the surrounding spatial context. However, existing methods—based on saliency—produce entangled explanations, making it unclear whether predictions rely on motion or spatial context. Language-based approaches offer structure but often fail to explain motions due to their tacit nature—intuitively understood but difficult to verbalize. To address these challenges, we propose Disentangled Action aNd Context concept-based Explainable (DANCE) video action recognition, a framework that predicts actions through disentangled concept types: motion dynamics, objects, and scenes. We define motion dynamics concepts as human pose sequences. We employ a large language model to automatically extract object and scene concepts. Built on an ante-hoc concept bottleneck design, DANCE enforces prediction through these concepts. Experiments on four datasets—KTH, Penn Action, HAA500, and UCF101—demonstrate that DANCE significantly improves explanation clarity with competitive performance. Through a user study, we validate the superior interpretability of DANCE. Experimental results also show that DANCE is beneficial for model debugging, editing, and failure analysis.

Disentangled Concepts Speak Louder Than Words: Explainable Video Action Recognition
Jongseo Lee'
Wooil Lee'
Gyeong-Moon Park
Seong Tae Kim Jinwoo Choilt
Kyung Hee University, Republic of Korea
Korea University, Republic of Korea
(jeng980812, 1wi2765, sim, jinwoochoi 8khu. kr, gm gn-parkkkorea.
Abstract
Effective explanations of video action recognition models should disentangle how movements unfold over time from the surrounding spatial context, However, ex- isting beseeds-baseed on saliency-produce entangled explanations, making it unclear whether predictions rely on motion or spatial context. Language-base approaches offer structure but often fail to explain motions due to their tacit nature-ine intuitivell understood but difficult to verbalize. To address these chal- lenges, we propose Disentangled Action aNd Context concept-based Explainable (DANCE) video action recognition, framework that predicts actions through disentangled concept types: motion dynamics, objects, and scenes. We define motion dynamics concepts as human pose sequences. We employ large lan- guage model to automatically extract object and scene concepts. Built on an ante-hoc concept bottleneck design, DANCE enforces prediction through these concepts. Experiments four datasets-KTH. Penn Action, HAA500, and UCF- 101-demonstrate that DANCE significantly improves explanation clarity with competitive performance. We validate the superior interpretability of DANCE through a user study. Experimental results also show that DANCE is beneficial for model debugging, editing, and failure analysis. Our project page available at https: /ttp:///oonnn///eeeee 12. git tuub.......................... io/DANCE/
Introduction
Recent advances video action recognition [56, 11, 28, 13, 32, 53, 30, 2] have led to impressive performance across diverse benchmarks. To deploy such high-performing video models in real-world applications, explaining their predictions becomes essential for ensuring trust, transparency, and accountability [10, 20]. Despite this need, the drcision-making processes of video action recognition models remain largely opaque, and systematic approaches explanation are still underexplored.
From a cognitive science perspective, humans interpret complex information more effectively when it is presented in structured format-i.e., broken down into meaningful and separable compo- nents [36, 34, 35, 51J. Interpretability further improves when each component expressed in a clear and unambiguous manner 10, 55, 33]. Notably, humans perceive actions by separately analyzing two distinct factors: (i) how movements evolve over time (temporal dynamics) and (ii) what phys- ical context surrounds those movements, such as objects and scenes (spatial context) [17, 16, 27]. Therefore, to align with human reasoning process, a video explainable Al (video XAI) should explain model's prediction
structured way may-explicitly disentangling and attributing its decisions to temporal dynamics and spatial context.
Meanwhile, existing approaches in video XAI largely follow strategies: extending image-based feature attribution methods [15, 23, 42, 14, 46] across the time axis [50, 18, 31] or clustering spatio- temporal tubelets to discover high-level concepts [26, 21, 43]. However, these methods do not
Corresponding authors.
39th Conference Neural Information Processing Systems (NeurIPS 2025).

---

Object concept
Scenee concept
"Busuball
"Bassbali field"
Besebblll
Little league
Tubea[so]
The person sctates their upper body <ienly
Motion dynamins concapt
the swing They fellow throuht by litinn the sar upoant a rurved motion, using thik antr and usper budy to ganelate faice"
VICDjas]
"Jorsey and field make senso. But molian? That's cacil?
"hha! New geet it!"
neis Bart listmer? Masion?
"Baseboi Slwing"
Example imput video
(8) Sellency-based explamation
(b) Langgagge-bbeee explanation
[6) DANCE explimetion
Figure 1: Disentangled concepts speak louder than words. Spatio-temporal attribution methods provide wnstructured explanations that are often ambiguous to human users. Given a Baseball Swing video, (a) visual explanations from 3D 3D-saliency [50] and VTCD [26] fail to clarify whether the prediction is driven by motion (e.g., torso twist), objects (e.g., jersey/helmet), or scene context (e.g., baseball field). (b) Language-based approaches offer more structure but remain ambiguous for motion, as is tacit knowledge- knowledge-intuttiveet intuitivvl understood but hard to verbalize. Verbal descriptions of motion often lack clarity and are difficult to interpret. (c) In contrast, DANCE disentangles motion and context to provide structured explanations. Pose sequences capture motion dynamics in an intuitive, appearance-invarian form, while we clearly convey object and scene concepts via text.
disentangle temporal dynamics from spatial context in their explanations. Instead, they highlight localized regions of input videos in an unstructured and entangled manner- manner-making it difficult to attribute predictions to specific types of evidence. For example, in Figure (a), for the action Basebull Swing, it remains unclear whether the model's prediction is based on the twisting motion of the torso, the appearance of the player (e.g., the jersey), or the scene (e.g., baseball field).
A potential direction for structured explanations is to use language-based approaches [38, 60, 39, 37, 1]. Language-based approaches could provide structured and human readable text descriptions, as illustrated in Figure (b). While these methods can effectively capture spatial context or high-level semantics, they often struggle to express motion dynamics clearly. This challenge arises because motion dynamics often fall under the category of tacit
that is intuitively
understood and applied, but difficult to verbalize or explain explicitly [41]. As shown in Figure (b), verbally describing the swinging motion of torso is nontrivial, and even when verbalized, such descriptions tend be overly verbose and cognitively difficult for users to interpret.
To address the challenges of structured and motion-aware explanation, we propose Disentangled Action aNd Context concept-based Explainable (DANCE) video action recognition framework. As illustrated in Figure (c), DANCE provides explanations based on three disentangled concept types: (i) motion dynamics, (ii) object, and (iii) scene. To capture fine-grained temporal patterns, we define motion dynamics concepts as human pose sequences. These pose sequences offer an appearance- agnostic representation of motion, enabling users to intuitively understand how an action unfolds over time without being distracted by irrelevant visual factors such as clothing or background. In parallel, we define object and scene concepts as action-related elements extracted using a large language model, allowing a model to incorporate spatial context concepts without manual annotation.
To ensure inherent explainability, DANCE adopts an ante-hoc design based on the concept bottleneck framework [25]. We insert concept layer between the backbone and the final classifier, enforcing the model to first predict concept activations before predicting the final action label. The concept layer comprises nodes for motion dynamics, object, and scene concepts. This disentangled design ensures that action predictions are explicitly grounded in both dynamic (pose sequence-based and static (object and scene) concept types. As result, explanations produced by DANCE are not only faithful to the model's reasoning but also well-aligned with human cognitive mechanisms.
To validate the effectiveness of DANCE, we conduct experiments on four video action recognition datasets: KTH [45], Penn Action [61]. HAA500 [9], and UCF-101 [49]. Results show that DANCE. significantly enhances explanation clarity by disentangling motion dynamics and spatial context, while maintaining competitive recognition performance against a model without interpretabilliyy A user study further demonstrates that explanations generated by DANCE are more faithful and interpretable compared to those of prior approaches. Extensive qualitative comparisons also highlight the superior structure and transparency of DANCE's explanations. Finally, we showcase the practical utility of DANCE across several downstream tasks- lickkuiinnddin model debugging, editing, and failure

---

case analysis underscoring its broader potential for understanding and improving video recognition models.
We summarize our major contributions as follows:
We propose DANCE, novel video XAI framework that provides structured and motion- aware explanations by disentangling motion dynamics and spatial context concepts.
We introduce a label-free pipeline that automatically discovers motion dynamics concepts via clustering of human pose sequences and spatial context concepts through LLM querying. The proposed pipeline allows DANCE. capture fine-grained motion patterns, objects, and scenes without manual annotations.
We conduct comprehensive evaluations across four datasets, assessing both explainability and performance through a user study, qualitative comparisons, and ablation studies. We further demonstrate the practical utility of DANCE in model debugging and editing.
2 Related Work
Video action recognition. The video action recognition task classifying human actions from temporally trimmed input video. Early approaches, such as two-stream CNNs [48], 3D CNNs [54, 12, 13], and temporal shift modules jointly encode spatial and temporal features to capture motion and context. More recently, transformer-based architectures [3, 40, 11, 53, 28, 2] have achieved substantial performance gains, largely due to large-scale pretraining. Despite these advances, the decision-making processes of most video action recognition models remain opaque, as predictions are made through complex, non-interpretable feature interactions. In this work, we propose DANCE that predicts actions based disentangled, numan-interpretable concepts, thus making the model's s reasoning process more transparent and well-aligned with human cognition.
Explainable video action recognition. Explaining the decision- making process of video action recognition models remains a relatively under-explored area. We can categorize existing methods into post-hoc explanation approaches- -uch as feature attribution [50, 31, 18j-and concept discovery methods [21, 26, 43]. These approaches typically use attribution optimization techniques to identify input regions (e.g., pixels or spatio-temporal tubelets) that contribute most to the model's prediction. As result, the explanations they produce are often unstructured and entangled, making difficult to attribute predictions to distinct types of reasoning, such as motion dynamics versus spatial context. Moreover, optimization-en based methods [21, 26, 43] require additional computational cost, as they need an optimization phase every time for generating explanation. In contrast, we propose an ante-hoc framework that explicitly disentangles temporal dynamics and spatial context concepts to produce structured, human-aligna explanations. Because DANCE is inherently explainable by design, can generate interpretable explanations in a single forward pass, without post-hoc optimization.
Dreentangled/eecomoseee explanations. Recently, there have been efforts to provide explanations disentangled into interpretable components the image domain [62, 24, 8, 25, 38, 60, 39, 47]. Concept bottleneck models (CBMs) insert a concept layer between the backbone and the final classifier, forcing predictions to be made explicitly through disentangled muman-interpretale cepts [25, 38, 60, 39, 47]. Other approaches disentangle what concepts influence the prediction and where the concepts occurs [52, 1, 4]. A separate line of work addresses the limitations of attribution- based explanations- entaangee attribution maps-by by disentangling intermediate-layed representations into concept subspaces [8] or selecting a compact set informative attribution regions [5]. While we also aim to provide disentangled explanations, our work differs in that we tackle the under-explored problem of explainable video action recognition. To the best of our knowledge, DANCE is a pioneer- ing work in video XAI by explicitly disentangling motion dynamics and spatial context concepts, enabling structured, interpretable, and human-ali -alinnd explanations for video model decisions.
con-
DANCE
We introduce DANCE,, an explainable video action recognition framework that produces structured and motion-aware explanations for its predictions. As illustrated in Figure 2, DANCE explains each prediction using three types of disentangled concepts: (i) motion dynamics, (ii) objects, and (iii) scenes. To capture fine-grained temporal patterns, we define motion dynamics concepts as representative human pose sequences extracted from training videos. These pose sequences offer appearance-agnostic representations of temporal motion, allowing users to clearly understand how an action unfolds over withhut being distracted by visual factors such as clothing, objects, or
3

---

cynamice condept
Motion
000.
Object
vidco bwckkooe r
Objjctt Concept Layer
"Beseball bat"
0.4
Baseboll Swing
Tonnis rockett
x.0.
Scene concept
Scene Concept Layyer
"Little loaguee field"
Gym
Ingut vidioo
Video teatures
Concept activations
Action prediction
R
& R
Figure 2: Overview of DANCE. Given an input video, DANCE first extracts video features using a pretrained video backbone encoder. Then, three disentangled concept layers project the video features onto their own concept space-motion dynamics, object, and scene-pree seene-producing disentangled activations. The interpretable classification layer linearly combines these concept activations predict the action class. By explicitly disentangling concept types, DANCE provides structured explanations that better align with how humans perceive actions by separating motion dynamics from the spatial context.
background. In parallel, identifying relevant objects and scenes associated with each action.
large language model (LLM) to extract spatial context concepts,
To ensure a transparent prediction, we adopt an ante-hoc design based on the concept bottleneck framework [25, 38]. As shown in Figure 2, insert concept layer between the backbone and the final classifier. Given a video, DANCE first predicts the activation of disentangled concepts, then uses these activations to produce the final action prediction. Through our disentangled concept bottleneck design, we ensure that explanation from DANCE structured and motion-aware. The remainder of this section is organized as follows. We introduce the concept bottleneck architecture Section 3.1, detail our concept discovery process in Section 3.2, present training procedures in Section 3.3.
3.1 Preliminary: Concept Bottleneck Model
Let us denote a training dataset as (V;, C. F.) where V, is the i-th input video, c; E {0, 13M a binary vector indicating the presence of M concepts, and y; E {0, 1)x a one-hot vector indicating the ground-trum action label among K classes. Given training sample (V,,c, we first extract a D-dimensional video-level feature vector x; f(V;) E IRD using a pre-trained video backbone encoder f()). We then project x, into M concept activations using linear concept layer g(:; Wc) parameterized by weights Wc RRDx z; g( x; Wc) R.M. Then, linear classifier h(: WA) with a softmax activation predicts an action label based on the concept activations: 9; = h(z; W E R
Unlike prior works [25, 38], DANCE explicitly disentangles three types of concepts- motion dynam- ies, objects, and scenes-to to provide structured and more intuitive explanations. To achieve this, we partition the parameters of the concept layer Wc into three disjoint parameters: Wc = [W W W:l. where W MD W E MMxD and W M, correspond to the parameters for motion dynamics, object and scene concepts, respectively. Here, Mn. M,, and M, denote the number of motion dynamics, object and scene concepts, respectively. We represent motion dynamics concepts using 2D human pose sequences, which explicitly capture how the human body moves over time in an appearance-agnon manner. For both object and scene concepts, we use intuitive text descriptions, e.g., baseball but, tennis court, that reflect the spatial context associated with each action.
3.2 Concept Discovery
For each concept type, first discover representative set of concepts using only the training videos from the target dataset. We then automatically annotate each video with the presence absence of these concepts, without requiring any human supervision.
3.2.1 Motion Dynamics Concept
As shown Figure 3 (a), we extract 2D pose sequences from all training videos and apply clustering to discover representative motion dynamics concepts. This allows us to build compact and interpretable vocabulary of movement patterns.
Key clip selection. In video sequences, not all clips are equally informative [22, 7, 6]; only a few temporally localized segments suchh wind-up, stride, release in bascball pitch-coniee

---

Traniiggast
Enample
Extraction
Pooe SEQuATIOS BY
Concept hand
Key
Poow estimatio
Clustering
Aggrepoing SeQuerIcE
Coneegt V: Oine downnwiing
Inpuc video
Key clip V/
Pose sequences of clips
Discovared motion dynamics concepts
(e) Motion dynemics concept discovery
Ma
Vileo Fronde
Nc.
Caroegt
Sin
Aui
Aee
(Q)
Mins
Prompt
out sreeoer
Alaratail
37
Re(3)
ritl
CIFE
Au
Encodar
entwidiigs gs Xy(0)
Siilartt
Gencopt
Contestlabd
Motion dynemics concept labeling
(c) Object concept discovery
(c) Object concept psoudo Labeling
Figure 3: Concept discovery and labeling process of DANCE. (a) Given a training video, we extract key clips with length L centered at keyframes identified by a keyframe detection algorithm. We then apply a 2D pose estimator to obtain human pose sequences from these key clips. By clustering all pose sequences across the training set, we cluster them define each cluster motion dynamics concept. (b) For each video, we derive binary motion dynamics concept labels by aggregating the cluster assignment tensor across its key clips. (c) To discover object concepts, we query GPT-4o [19] with prompts containing action class names, yielding set of object concepts for the dataset. (d) Given video and the object concept set, we compute concept pseudo labels using vision-language dual encoder. Specifically, we obtain concept pseudo label vector by multiplying the object concept embedding matrix with the video embedding vector. We can obtain scene concept labels analogously.
distinctive motion cues critical for action recognition. To focus on such informative segments, we extract pose sequences from key clips only. We first detect keyframes by rumning an off off-the-shelf method" using pixel value differences. For each selected keyframe, we extract short video clip V; of fixed length L centered at that frame, where s E {1...... .S} is the key clip index. This targeted sampling strategy allows us to concentrate the concept discovery process on primitive, discriminative motion patterns that are frequently shared across different instances of the same action class and in some cases, across classes. Please refer to the supplementary materials.
Pose sequence extraction. For each key clip Vi, we apply 2D pose estimation model [59] to every frame to obtain a pose sequence P; E x2xxxx2 where / is the number of joints. To ensure high-quality motion dynamics representations, large discontinuities in joint coordinates between consecutive frames. For further implementation details, please refer to the supplementary materials.
filter out pose sequences with low average joint confidence or
Concept discovery. To discover motion dynamics concepts, we first aggregate all pose sequences from the training videos into unified set: P = , P. where P denotes the pose sequence from the s-th key clip of the i-th video. To group similar motion patterns, we apply clustering algorithm, e.g., FINCH [44], to the aggregated set P, as illustrated in Figure 3 (c). We flatten each pose sequence into feature vector before clustering. We define each resulting cluster
distinet motion dynamics concept and assign it unique concept index E {I,.... ......., Mn). where Mn, is the total number of motion dynamics concepts. Based on the clustering results, we construct binary cluster assignment tensor = [aisa]] E 1] Nx5xM, where each element a,,,, is defined as:
fI,
if P. belongs to cluster k, 0, otherwise.
4|,s
Concept labeling. For each training video V,, we assign motion dynamics concept labels by checking whether any of its pose sequences P, belong to given cluster k:
C T( aiax);
(2)
5

---

where I(-) is the indicator function. As result, we obtain a binary motion dynamics concept label vector e."h E {0, 1, indicating which motion dynamics concepts are present in video V,. Note that this labeling process is entirely unsupervised, requiring no manual annotations.
3.2.2 Object and Scene Concept
Concept Discovery. To discover intuitive and clear concepts in an unsupervised manner, we leverage large language model, as illustrated in Figure 3 (d). For each action class, we query GPT-4o [19] with two prompts: i) "For the <action class>, list the most important physical objects that commonly appear when this action occurs." and ii) "List the most common places or background scenes where <action class> typically occurs. Do not include objects or equipmen" " These prompts yield diverse and semantically meaningful set of candidate object and scene concepts associated with each action. To improve concept quality and reduce redundancy, we follow prior work [38] by applying post-prooe pos-pproessiig filters removing overly long phrases, near-duplicates, and concepts that are overly similar to the action class name. For more implementation details and examples, please refer the supplementary materials.
Concept Pseudo Labeling. To avoid manual concept annotation, we employ a rision-language dual encoder [57] to generate concept pseudo labels for each training video V. Let (-) and Ey (-) denote the video and text encoders of the dual encoder, respectively. We first obtain video embedding vector Ev (V;) E IRD, where D is the shared embedding dimension. We then encode the object concept set O obtained from GPT-4o using the text encoder to obtain an object concept embedding matrix E, (0) E MMx where M,, denotes the number of object concepts. Given object concept embeddings Er O) and the video embedding Ev (V.). we compute the object pseudo concept label vector ες as:
ن = Er (O) E [0,1]*.
Note that हi is soft label. We can obtain scene concept labels ह E [0, 1J analogously using the scene concept embedding matrix Ey (S) R2 and the video embedding Ey (V.). For more details, please refer the supplementary materials.
3.3
Training
We freeze the pretrained video backhone encoder f(-) and train the concept layer g(: Wc) and the final classification layer h(:; WA) ) in separate stages, using concept labels derived in Section 3.2. For brevity, we omit the batch dimension and the sample index in the following descriptions.
Motion dynamics concept layer. Since the motion dynamics label c." {0, 1MMr, derived from (2), is a multi-label binary vector rather than one-hot vector, we train the motion dynamics concept parameters using the binary cross-entropy loss. Given the motion dynamics concept activations z" gx; Wi),, we apply the sigmoid activation (-.) to each element and define the loss as:
Man Σ
Lm
|e log[oo (z")) (1- CTH) log(( o2 (z"
Mm
Here, (z")) denotes the k-th element of (z"").
Object and scene concept layer. To train the object concept layer, we follow prior work [38] and apply the cosine cubed loss between the pseudo label vector E (derived from (3)) and the object concept activations z" g(x: Wy). The cosine cubed loss emphasizes directional alignment between the predicted and target concept representations while being invariant to scale. For additional details, we refer readers to the methodology described in [38]. We train the scene concept layer in an analogous manner using the corresponding pseudo labels and activations.
3
L,
(5)
203
2
Interpretable classifier. We freeze the learned concept layers and train a final linear classifier that predicts actions solely based on the concept activations. By enforcing predictions to be made through disentangled and interpretable concepts, we promote transparent and structured explanations. Let z= z"="=" denote the concatenated motion dynamics, object, and scene concept activations. We train the classifier using the standard cross-entropy loss between the ground-truth action label y and

---

Penn Action datoset
KRAA
plrchert recund
Socetall Pizh
video
GradCAM(41)]
Vatageaa
Bench
Cinncerr cscrrbutiinn
Saliency Tubes[50]
Velleg Poike
Object:
Tennis
Paece
DANCA
explanation
Soliencc--basedd explanation
Basketbail Shoot Volleyboll Set
Bosebolll
Serve
Figure 4: Sample-level explanation. We com- pare (a) DANCE with (b) existing attribution- based methods: GradCAM [46] attribution and Saliency Tubes [50].
Figure 5: Visualization of model weights of sim- ilar action class pairs. For each example, we show Sankey diagram of the final layer weights associated with a similar action class pair.
Which explains better?
Expert Knowledge
Language-based
Ours
GPT
IT
Ours
Knowtodge
Peroantage responses
Ang
Soure 3.44
Avg Soure :2.3
(9) Interpretability of DANCE
(b) Interpretability ofe motion dynomics concept
Figure 6: Interpretability of DANCE. (a) We present user study results from pairwise comparisons evaluating the interpretability of DANCE against three video XAI baselines: (i) a concept bottleneck model using spatio-temporal concepts generated by GPT-4o [19], (ii) VTCD [26], a spatio-temporal saliency-based explanation method, and (iii) a concept bottleneck model using spatio-temporal concepts from UCF-101 attributes [49]. (b) We report user study results comparing the interpretability of three concept types: (i) inguage-based concepts generated by GPT-4o, (ii) expert-defined concepts based on UCF-101 attributes [49], and (iii) our proposed motion dynamics concepts.
the action prediction h(z; WA), with regularization term to enhance interpretability [58, 38]:
Lets Σ ya log((())) + a) Wall+ [|all,,,,
(6)
where &a denotes the k-th element y, |1-11= represents the Frobenius norm, I is the element-wise & norm, and 2 and are balancing hyperparameters.
4 Experimental Results
In this section, carefully design and conduct rigorous experiments to answer the following research questions: (1) Does DANCE generate explanations that are easy for humans interpret the context of action prediction? (Section 4.1) (2) Can DANCE detect changes in the temporal domain, such as reversed input sequences? (Section 4.1) (3) What is the performance trade-off, if any, when interpretability is introduced into previously
-nteeeerrreeeb model? (Section 4.2) (4) Can DANCE be effectively used for model debugging and editing? (Section 4.3) For more details on the dataset and implementation, please refer to the supplementary materials.
4.1 Analysis
In this section, we examine whether DANCE produces explanations that are easily interpretable for humans in the context of action prediction. To this end, first define concept contribution to quantify the importance of each concept neuron, and then provide sample-level and model-level explanations, followed by the results of a user study evaluation. Additional visualizations and further details for this section provided in the supplementary materials.
Concept contribution. We define the concept contribution as the product of a concept activation and the concept weight associated with the predicted class and we use this term consistently throughout the paper. To visualize motion dynamics concept, we select the pose sequence closest to the cluster medoid and use it as the representative example.
7

---

1st contributed concept
2nd contributed concept
GT: Gym Plank
Pred:
Burpee
Backward Video
Iidut video
Pred: : Push up x
Concepts
1.3
Original Bowing Fullbody Video
Pred:
Bowing
FullBody
1s contributed concept
2nd contributed concept
Concept contribution
Figure 7: Sanity check of DANCE. We compare predic- tions and top-2 contributing concepts of DANCE for (i) the original video, and (ii) the same video played backward.
Failure case analysis using DANCE. With intuitive explanations, DANCE can help failure case analysis.
Figure 8:
Sample-level explanation. In Figure 4, we visualize the top-3 contributing concepts along with the input video. In Figure 4 (a), DANCE leverages the motion concepts "lowering" and "lifting", along with the object concept "Bench", which together support the correct prediction. In contrast, as shown in Figure 4 (b), saliency-based methods [46, 50] produce spatio-temporrall entangled explanations, making it unclear whether the model bases its decision on the object itself, i.e., "barbell" or its motion, i.e. up-down movement when predicting Bench Press.
Model-level explanation. In Figure 5, we visualize concept-to-clan weights for pairs of similar action classes using Sankey diagrams. The thickness of each edge represents the relative contribution weight. DANCE provides the model's s decision basis, helping users understand how it discriminates between similar actions. For example, in Figure 5 (a), Basketball Shoot and Volleyball Ser share common motion concepts (shown in light khaki), while DANCE distinguishes between them using subtle motion differences (shown in pink) and the object concept, "Volleyball Poles."
Interpretability of DANCE. To evaluate the effectiveness of DANCE in explaining model pre- dictions, we conduct a user study comparing it against three baseline explainable video action recognition methods: (i) CBM using entangled opatio-temporal concepts generated by GPT-4o [19], (ii) VTCD [26], a spatio-temporal saliency- method, and (iii) a CBM using spatio-temporal con- cepts from UCF-101 101 attributes [49]. In each question, we ask each participant a pairwise comparison question, by showing a video along with explanations from DANCE and one of the baselines: "Which explanation helps you better understand why the model predicted the action?" We collect responses on a five-point Likert scale, ranging from "DANCE is much better" to "the other method is much better." As shown in Figure (a), more than 70% of the responses fall into "ours is much better" or "ours is slightly better" categories across all three pairwise comparisons. These results showcase that users perceive DANCE as more intuitive and trustworthy than (i) language-basee explanations, (ii) unstructured saliency-basee based attributions, and (iii) expert-defined concepts. For more details, please refer to the supplementary materials.
Interpretability of the proposed motion dynamies concept. We conduct a user study evaluating the interpretability of three CBMs using the following temporal concepts: (i) language- based concepts generated by GPT-4o [19], (ii) expert-defined concepts based on UCF-101 attributes [49], and (iii our proposed motion dynamics concepts. For each baseline, we randomly select one concept and visualize the top six video clips whose top-1 concept activation corresponds to the selected concept. We then presented the concept alongside the activated video clips, and asked participants the following question: "How well does the given concept match the actions or motions shown in the video?" We collect the response on five-point Likert scale, where higher scores indicate stronger perceived alignment and interpretability. As shown in Figure 6 (b), our motion dynamics concept achieves the highest average score of 4.3, with 89.7% of participants rating it 4 or 5. In contrast, language-bal -basd based and expert-defined concepts receive lower average scores of 2.3 and 3.4, respectively. These results indicate that the proposed motion dynamics concept is significantly more intuitive and aligned with human perception of motion, providing more interpretable explanations. For more details, please refer to the supplementary materials.
baaed
Sanity check. Here, we check the sanity of DANCE from the perspectives of model behavior and explanation quality. In Figure 7, we compare the predictions and top-2 contributing concepts of both methods for (i) the original video and (ii) the same video played backward. DANCE correctly predicts Bowing FullBody for the original video leveraging the visualized motion dynamics concepts. For the backward video, DANCE predicts Burpee by leveraging "standing up" like motions as visualized,
8

---

Table Video action recognition performance. We report the Top-1 accuracy (%) of the baselines with and without interpretability as well as DANCE, all using the same backbone encoder [53].
Method
KTH [45]
Penn Action [6]]
HAA- 10A-10009) [9]
UCFFIDI [49]
Baselino interpeetaiiiity
89.7
97.8
88.4
CBM [25] w/ UCFF11) attributes
86.8
LF-CBM [38] w/ entangled language concepts
87.4
96.3
85.5
LF-CBM [38] w/ discntangled language concepts
89.9
97.7
65.3
83.7
DANOB
91.1
98.1
70.7
87.5
Or-Cricket
Domain shift example
Iidut Video Concepts
Golf Swing
Toble Tennis Club
0.0
In-domain
Cross-domain video
Desctivatin!
Gotf
Valleyboll Spiling
Tanril Swing
1.6
FFFR concept
AAAL. Ka
%
concept
Concept
#5.m9 87.0% temples
64.0% y 86.5
comples
82.2% B4.1%
samplex taramppl
Table Tennis
Original Pred:
New Pred:
(+1.4poimts)
eammiin
(+2.5 points)
Exmpii
(+1.9 poires)
Joint Intervention:77..7 7% 82.0% (+4.3 points)
Figure 9: Sample-leve interven- tion. Deactivating the irrelevant concept leads to fixing the mis- prediction to a correct predcition.
Figure 10: Cross domain class-level intervention. Fixing zero weights from relevant concepts to classes result in 4.3 point accuracy improvement under severe domain shift, i.e., UCF- 101 [49] + UCF-1O1-SCOO 101-SCUBA [29], without retraining.
reflecting the model's 's sensitivity to temporal direction. With the sanity check, we validate that (i) the model is sensitive to motion dynamics as intended, and (ii) explanations by DANCE are interpretable. Failure case analysis. Thanks to the transparent and intuitive nature of DANCE, we can also analyze the reason behind incorrect predictions as shown in Figure 8, DANCE explains that the misprediction of Push up is because of high contributions from downward body movement. Such interpretability enables targeted model debugging, as further demonstrated in Section 4.3.
4.2 Performance Evaluation
Interpretability and performance do not always trade-off. We compare DANCE with a baseline model without interpretability, as shown in Table 1. DANCE achieves slightly higher accuracy KTH and Penn Action, while showing a modest drop of 2.8 points on HAA-100 and 0.9 points on UCF-101. These results indicate that DANCE can deliver intuitive and structured explanations without substantially compromising classification performance-
in some cases, even improving it.
Clearer concepts improves performance. We further investigate whether the clarity of concept representations affects model performance. To this end, we compare DANCE with the following baselines: (i) concept bottleneck model with UCF-101 attributes [49], (ii) a label-free concept bot- tleneck model [38] using spatio-temporally entangled concepts generated by GPT-4o, and (iii) a variant of the label-free concept bottleneck model that uses the same object and scene concepts as DANCE, but uses SGPT-4o-generatee temporal concepts. Across all datasets, DANCE consistently outperforms these baselines, demonstrating that employing clearer and disentangled concept repre- sentations- particularry for motion dynamics can lead improved action recognition performance.
4.3
Model Editing
Here, we demonstrate the utility of DANCE in debugging itself by analyzing sample-level concept contributions in misclassifications and inspecting class-level weights. We provide additional results of model editing in the supplementary materials.
Sample-level intervention. In Figure 9, we illustrate how a user can intervene the model by removing a specific concept in the case of misclassification. For example, in Figure 9 (a), the model initially predicts Table Tennis Shot, largely influenced by the scene-level concept "Table tennis club." " When this concept is deactivated, the model leverages motion dynamics concepts and correctly classifies the input Cricker Shot. The results demonstrates that DANCE supports fine-grained transparent control over predictions, allowing users to actively adjust model behavior.
Cross domain class-level intervention. In Figure 10. we demonstrate class-level intervention using DANCE. This experiment evaluates whether such adjustments can resolve performance drop caused by severe distribution shift. We evaluate on UCF-10I-SCUBA [29], a variant of UCF-101 [49] where test video backgrounds are altered to induce domain shift. As shown in Figure 10, for the

---

Volleyball Spiking class, the model activates relevant motion dynamics concept but ignores it due to zero weight. By assigning weight of 1.0 to this concept, we correct 98 misclassifictiinn with only one additional error, significantly improving overall accuracy by 2.5 points (84.0% - 86.5%). Further adjusting weights for the Golf Swing and Tennis Swing classes result in 4.3 point accuracy improvement (77.7% - 82.0%). These findings highlight DANCE's ability to support post hoc model debugging and performance recovery under severe domain shifts, without retraining.
Conclusions
In this paper, we propose DANCE to address the challenge of explaining video action recognition models in a structured and motion-aware manner. DANCE grounds its predictions in three human- interpretable concept types-motion
motion dynamics, objects, and ceaneinnnablin cognitively aligned and transparent explanations. This design facilitates intuitive understanding of model behavior by explicitly separating temporal and spatial reasoning. Through extensive experiments and practical use cases, we show that DANCE. delivers clearer and more faithful explanations while maintaining competitive recognition performance. Moreover, DANCE supports effective model editing even under severe domain shifts shifts-without without requiring retraining. We believe our work offers valuable insights to the XAI and video understanding communities and will help inspire future research.
Acknowledgment. This work was supported in part by the Institute of Information & Communi- cations Technology Planning & Evaluation (ITTP) grant funded by the Korea Government (MSIT) under grant RS- RSS2024-00353133 (20%).. RS-2021-0-00000 20%), and AS-2022-00153)) Development (Kyung Hee University, 20%)), Additionally, it was supported by the National Research Foundation of Korea(NRF) grant funded by the Korea and (RS-2025- 2---2002522366 2236296,, 20%). We thank Gangmin Choi, Kyungho Bae and Yong Hyun Ahn for the valuable discussions and feedback.
(Artificial Intelligence Innovation Hub, (Artificial Intelligence Convergence Innovation Human Resources
20%)
References
[1] Yong Hyun Ahn, Hyeon Bac Kim, and Seong Tae Kim. Www: unified framework for explaining what where and why of neural networks by interpretation of neuron concepts. In CVPR, 2024. 2,
[2] Kyungho Bae, Geo Ahn, Youngrae Kim, and Jinwoo Choi. Devias: Learning disentangled video represen- tations of action and scene for holistic video understanding. In ECCV, 2024. 1,3
[3] Gedas Bertasius, Heng Wang, and Lorenzo Torresani. Is space-time attention all you need for vidco understanding? In ICML, 2021. 3
[4] Chaofan Chen, Oscar Li, Daniel Tao, Alina Barnett, Cynthia Rudin, and Jonathan K Su. This looks like that: deep learning for interpretable image recognition. In NeurlPS, 2019. 3
[5] Ruoyu Chen, Hua Zhang, Siyuan Liang, Jingzhi Li, and Xiaochun Cao. Less is more: Fewer interpretable region via submodular subset selection. In ICLR, 2024. 3
[6] Yifci Chen, Dapeng Chen, Ruijin Liu, Hao Li, and Wei Peng. Video action recognition with attentive semantic units. In ICCV, 2023. 4
[7] Jinwoo Choi, Gaurav Sharma, Samuel Schulter, and Jia- Bin Huang. Shuffle and attend: Video domain
adaptation. In ECCV, 2020. 4
[8] Pattarawat Chormai, Jan Herrmann, Klaus-Robert Müller, and Grégoire Montavon. Disentangled explana- tions neural network predictions by finding relevant subspaces. TPAMI, 2024. 3
[9] Jihoon Chung, Cheng-hsin Wuu, Hsuan-ru Yang. Mo-Wing Tai, and Chi- Chi-Keung Tang. Haa500: Human- centric atomic action dataset with curated videos. ICCV, 2021. 2, 9, 18
[10] Finale Velei and Been Kim. Towards rigorous science of interpretable machine leaming. arXiv preprint X00. 08608. 2017.1
[11] Haoqi Fan, Bo Xiong, Karttikeya Mangalam, Yanghao Li, Zhicheng Yan, Jitendra Malik, and Christoph Feichtenhofer. Multiscale vision transformers. In ICCV, 2021. 1,
[12] Christoph Feichtenhofer. X3d: Expanding architectures for efficient video recognition. In CVPR, 2020.
[13] Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recognition. In ICCV, 2019. 1.3
10

---

[14] Ruth Fong, Mandela Patrick, and Andrea Vedaldi. Understanding deep networks via extremal perturbations and smooth masks. In ICCV, 2019.
[15] Amirata Ghorbani, James Wexler, James Y Zou, and Been Kim. Towards automatic concept-based explanations. In NeurlPS, 2019. 1
[16] Melvyn A Goodale and David Milner. Separate visual pathways for perception and action. Trends in neurosciences, 15(1):20-22 25, 1992.
[17] Emily Grossman, Michael Donnelly, R Price, Pickens, V Morgan, Neighbor, and Randolph Blake. Brain areas involved in perception of biological motion. Journal of cognitive neuroscience, 12(5),711-75 2000.
[18] Thomas Hartley, Kirill Sidorov, Christopher Willis, and David Marshall. Swag-v: explanations for video using superpixels weighted by average gradients. In WACV, 2022. 1,3
[19] Aaron Hurst, Adam Lerer, Adam P Goucher. Adam Perelman, Aditya Ramesh, Aidan Clark. AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint 27.2:224 2024. 6, 7, 8, 18
[20] Alon Jacovi, Ana Marasović, Tim Miller, and Yoav Goldberg. Formalizing trust in artificial intelligence: Prerequisites, causes and goals of human trust in ai. In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency, 2021.
[21] Ying Ji, Yu Wang, and Jien Kato. Spatial-temporal concept based explanation of 3d convnets. In CVPR, 2023. 3
[22] Amlan Kar, Nishant Rai, Karan Sikka, and Gaurav Sharma. Adascan: Adaptive scan pooling in deep convolutional neural networks for human action recognition in videos. In CVPR, 2017.
[23] Been Kim, Martin Wattenberg, Justin Gilmer, Carric Cai, James Wexler, Fernanda Viegas, et Inter- pretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). In ICML, 2018.
[24] Lukas Klein, João BS Carvalho, Mennatallah El-Assady. Paolo Penna, Joachim M Buhmann, and Paul F Jaeger. Improving explainability of disentangled representations using multipath-attribuiie mappings. In MIDL, 2022.
[25] Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. Concept bottleneck models. In JCML, 2020. 2, 9
[26] Matthew Kowal, Achal Dave, Rares Ambrus, Adrien Gaidon, Konstantinos G Derpanis, and Pavel Tok- makov. Understanding video transformers via universal concept discovery. In CVPR, 2024. 1, 2, 3, 7,
[27] Dwight J Kravitz, Kadharbatcha S Saleem, Chris Baker, and Mortimer Mishkin, A new neural framework for visuospatial processing. Nature Reviews Neuroscience, 12(4))217-230, 2011.
[28] Dongho Lee, Jongseo Lee, and Jinwoo Choi. Cast: Cross-attention in space and time for video action recognition. In NeurlPS, 2023. 1,3
[29] Haoxin Li, Yuan Liu, Hanwang Zhang, and Boyang Li. Mitigating and evaluating static bias of action representations in the background and the foreground. In /CCV, 2023. 9
[30] Kunchang Li,, Xinhao Li, Yi Wang. Yinan He, Yali Wang. Limin Wang, and Yu Qiao. Videomamba: State space model for efficient video understanding. In ECCV, 2024. 1
[31] Zhengiang Li, Weimin Wang, Zuoyue Li, Yifei Huang, and Yoichi Sato. Towards visually explaining video understanding networks with perturbation. In WACV, 2021. 1.3
[32] Ji Lin, Chuang Gan, and Song Han. TSM: Temporal Shift Module for Efficient Vidco Understanding. In ICCV, 2019. 3,
[33] Zachary C Lipton. The mythos model interpretability: machine learning, the concept of interpretability is both important and slippery. Queue, 16(3):31-57, 2018.
[34] Richard E Mayer. Evidence-based principles for how to design effective instructional videos. Journal of Applied Research in Memory and Cognition, 10(2):229-24) 2021.
[35] Richard E Mayer and Roxana Moreno. Nine ways reduce cognitive load in multimedia learning. Educational psychologist, 38(1):43-53 52, 2003.
11

---

[36] Richard E Mayer and Celeste Pilegard. Principles for managing essential processing in multimedia learning: Segmenting, pretraining, and modality principles. The Cambridge handbook of multimedia learning, 13: 316 44, 2005.
[37] Tuomas Oikarinen and Tsui- Wei Weng. Clip-dissect: Automatic description of neuron representations in deep vision networks. In ICLR, 2023. 2
[38] Tuomas Oikarinen, Subhro Das, Lam Nguyen, and Lily Weng. Label-free concept bottleneck models. In ICLR, 2023. 2, 3, 4, 7,9
[39] Konstantinos Panousis, Dino Ienco, and Diego Marcos. Coarse-to-fine concept bottleneck models, In NeurIPS, 2024. 3,
[40] Mandela Patrick, Dylan Campbell, Yuki Asano, Ishan Misra, Florian Metze, Christoph Feichtenhofer, Andrea Vedaldi, and Joao F Henriques. Keeping your eye on the ball: Trajectory attention in video transformers. In NeurlPS, 2021. 3
[41] Arthur S Reber. Implicit learning and tacit knowledge. Journal of experimental psychology General, 118 (3)2219, 1989. 2
[42] Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. Why should trust you? explaining the predictions of any classifier. In KDD, 2016.
[43] Avinab Saha, Shashank Gupta, Sravan Kumar Ankireddy, Karl Chahine, and Joydeep Ghosh. Exploring explainability in video action recognition. In CVPRW, 2024. 1,3
[44] Saquib Sarfraz, Vivek Sharma, and Rainer Sticfelhagen. Efficient parameter-frce clustcring using first neighbor relations. In CVPR, 2019, 5
[45] C. Schuldt, I. Laptev, and B. Caputo. Recognizing human actions: local svm approach. In ICPR, 2004. 2.
9, 18
[46] Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In ICCV, 2017. 1,7.8
[47] Chenming Shang, Shiji Zhou, Hengyuan Zhang, Xinzhe Ni, Yujiu Yang, and Yuwang Wang. Incremental residual concept bottleneck models. In CVPR, 2024.
[48] Karen Simonyan and Andrew Zisserman. Two-stream convolutional networks for action recognition in videos. In NeurlPS, 2014. 3
[49] Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. dataset human action classes from videos in the wild. Center for Research Computer Vision, 2(11):1-7, 2012. 2, 7, 8, 9, 18
[50] Alexandros Stergiou. Georgios Kapidis, Grigorios Kalliatakis, Christos Chrysoulas, Remco Veltkamp, and Ronald Poppe. Saliency tubes: Visual explanations for spatio-temporal convolutions. In ICIP, 2019. 1,
3, 7,
[51] John Sweller. Cognitive load during problem solving: Effects on learning. Cognitive science, 12(2): 257-285, 1988.
[52] Fel Thomas, Picard Agustin, Bethune Louis, Boissin Thibaut, Vigouroux David, Colin Julien, Cadène Rémi, and Serre Thomas. Craft: Concept recursive activation factorization for explainability. In CVPR, 2023. 3
[53] Zhan Tong, Yihing Song, Jue Wang, and Limin Wang. VideoMAE: Masked autoencoders are data-cfficient learners for self-supervised video pre- pre-training. In NeurlPS, 2022. 1, 18
[54] Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri, Learning spatiotemporal features with 3d convolutional networks. In ICCV, 2015.
[55] Barbara Tversky, Julie Bauer Morrison, and Mireille Betrancourt. Animation: can it facilitate? International journal of comann-compuer studies, 57(4)2247-262 2002. 1
[56] Rui Wang, Dongdong Chen, Zuxuan Wu, Yinpeng Chen, Xiyang Dai, Mengchen Liu, Lu Yuan, and Yu-Gang Jiang. Masked video distillation: Rethinking masked feature modeling for self-supervised video representation learning. CVPR, 2023.
[57] Yi Wang, Yinan He, Yizhuo Li, Kunchang Li. Jiashuo Yu, Xin Ma. Xinhao Li, Guo Chen, Xinyuan Chen, Yaohui Wang, et al. Internvid: A large-scale video-text dataset for multimodal understanding and generation. In ICLR, 2023. 6
12

---

[58] Eric Wong, Shibani Santurkar, and Aleksander Madry. Leveraging sparse linear layers for debuggable deep networks. In /CML, 2021. 7
[59] Yufei Xu, Jing Zhang, Qiming Zhang, and Dacheng Tao. Vitpose: Simple vision transformer baselines for human pose estimation. In NeurlPS, 2022. 5, 18
[60] Mert Yuksekgonul, Maggie Wang, and James Zou. Post-hoc concept bottleneck models. ICLR, 2023.
[61] Weiyu Zhang, Menglong Zhu, and Konstantinos G. Derpanis. From actemes action: A strongly- supervised representation for detailed action understanding. In ICCV, 2013. 2, 18
[62] Yingji Zhang, Danilo S Carvalho, and André Freitas. Learning disentangled semantic spaces of explanations via invertible neural networks, arXiv preprint 2rX1v:2305.01713, 2023, 3
13

---

1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the
paper's contributions and scope?
Answer: [Yes]
The main claims in the abstract and introduction accurately summarize the contributions and scope of the paper. The claims are supported by experimental results, as detailed in section 4.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings. is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: The limitations our work are thoroughly discussed in the dedicated "Limita- tions" section of supplementary materials.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faireess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification: The paper does not include any theoretical results.
14

---

Guidelines:
The answer NA means that the paper does not include theoretical results,
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
The paper provides all necessary details to reproduce the main experimental results, including detailed descriptions of the dataset, model architecture, and evaluation metrics. These are thoroughly documented in the supplementary material.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome deseribe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
15

---

Justification:
We provide code in the supplementary material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https: //ss:////ss.//////p//// cc/
public /uwidduiisssseeeeee / Code ssi ssssoPolo (onoollcy)) for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
n ps cc/ ptpppplllsc c/ i des /Cooeeoouubao ss onPol 1 ccy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
The paper specifies all the necessary training and test details, including data splits, hyperparameters, and the type of optimizer used. These details are presented the supplementary materials.
Justification:
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: While the paper provides some information about the experimental results, it lacks detailed reporting of error bars or other statistical significance measures. This may affect the interpretation of the results. Future revisions will aim to include these details.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
16

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
bars, but should state it. The authors should
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are of range (e.g. negative error rates).
If error bars are reported in tables or plots. The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
Question: For each experiment, does the paper provide sufficient information the computer resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Answer: [Yes]
Justification: We report compute resource details GPU type, memory, and per-experiment training time-in the supplementary materials. This includes estimates of training time for each dataset, hardware specifications, ensuring reproducibility.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: http://////eii....eee http://cccc//tttcccccleelllllll.
Answer: [Yes]
Justification:
The research conducted in this paper fully conforms with the NeurIPS Code of
Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
We include diseussion of potential broader impacts in the supplementary materials. Specifically, we highlight the positive societal impact of improving the inter- pretability of video action recognition models by disentangling motion dynamics from spatial context-enhancing transparency and trust in applications such as healtheare, ed- ucation, and safety-critical systems. We also acknowledge potential risks, such as misuse in surveillance scenarios, and emphasize the importance of responsible deployment and ongoing monitoring.
Justification:
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
17

---

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper does not involve the release of data or models with high risk for misuse, hence this question is not applicable.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We use publicly available datasets (UCF-101 [49], Penn Action KTH [45],
HAA-500 [9]) under their academic licenses. We also the pre-trained VideoMAE [53] encoder (Apache 2.0 license) and the ViTPose (Apache 2.0 license) pose estimator [59]. For
object/scene concept generation, we query GPT-4o [19], which is used in accordance with OpenAl's usage policy. All sources are cited, and licenses are respected.
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
18

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, papersw thcode com / dat sets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We introduce new implementation of the DANCE framework, including code for concept discovery, training, and visualization. If released, the code will include documentation, usage instructions, and evaluation scripts. We also curate a subset of the HAA-500 dataset (HAA-100) to facilitate interpretable evaluation. The curation process and selection criteria are clearly described in the supplementary material.
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsoureing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [Yes]
Justification: The paper includes a user study with human subjects to evaluate the inter- pretability of soncept-based explanations. Key methodological details are provided in the main paper. The supplementary material includes full task instructions, interface screenshots, and compensation information.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based the requirements of your country or institution) were obtained?
Answer: [Yes]
Justification: The study involving human subjects was reviewed by an Institutional Re- view Board (IRB) and received exemption determination based minimal risk and anonymized data collection. No personally identifiable information was collected, and all participants were informed and gave consent prior to participation.
19

---

Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality the research, declaration is not required.
Answer: [Yes]
Justification: We employed a large language model (LLM), specifically GPT-4o, to gener- ate candidate object and scene concepts that form part of concept vocabulary. These LLM -eneertdd concepts serve as interannittrrettablee descriptors used in our explanation framework. The LLM was not involved in model training or prediction but contributed to the construction of interpretable spatial concepts, which are integral our method.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy (https: //tpuriiss.p///// cc/ wwc//oofeeeencee.///ooeee /2055/ LLM) for what should should not described.
20


Reviews:
[{"review_id": "wsIzpFXLxd", "reviewer": "Reviewer_nxgA", "summary": "- This work proposes Disentangled Action and Context concept-based Explainable (DANCE) video action recognition\n    \n- The DANCE framework uses a frozen video backbone to extract features, then projects these features into a set of interpretable object, scene, and motion dynamics activations\n    \n- These interpretable concept activations are used as features for a linear action prediction layer\n    \n- To obtain a set of discrete motion dynamics concepts, they extract 2D pose sequences with a pose estimation model from the training videos and cluster the pose sequences. A video’s poses are then represented as a binary multi-label vector indicating which clusters the video’s pose sequences are members of\n    \n- To obtain object and scene concepts, they prompt an LLM to list the likely objects and scenes associated with each action class\n    \n- Training proceeds by first training the projection layers to predict the motion dynamics, then training a linear layer on the predicted concepts\n    \n- They find that DANCE’s explanations are heavily preferred to other interpretable baselines, and that classes are readily interpretable based on highly-weighted concepts\n    \n- The model’s interpretability allows for straightforward model editing, and they show that this process can improve classifications under distribution shift", "questions": "Please see the weaknesses for changes I would like to see addressed (providing more context of DANCE's performance relative to baselines).\n\nFor clarification: It is unclear what “when this concept is deactivated” means on line 314. Is it assigned a weight of zero?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "# Strengths\n\n- Straightforward, effective approach to an intrinsically interpretable action recognition model\n    \n- Results show a clear preference for their model over saliency-based methods and concept bottleneck models with spatio-temporal concepts, showing the benefit of pose-based interpretable features\n    \n- The writing is clear and the figures informative\n    \n- Qualitative examples are convincing of the approach’s explainability\n    \n- Manual model editing to avoid complete model retraining is a useful capability of this approach\n    \n\n# Weaknesses\n\n- The non-interpretable baseline in Table 1/line 296 is not described. How comparable is this model? Is it finetuned, zero-shot? [PapersWithCode](https://paperswithcode.com/sota/action-recognition-in-videos-on-ucf101) suggests that the top performance on UCF101 is 99.70, for example—significantly higher than the performance of the baseline w/o interpretability’s 88.4%. It is reasonable to focus on appropriate comparisons to DANCE, but more context for the baseline and SOTA models should be provided\n    \n- This method trades off flexibility for the sake of interpretability—adding a new class would likely require retraining the entire pipeline (e.g. mining concepts and clustering from scratch). A flexible but opaque alternative would simply be to use an open-vocabulary model. An explicit comparison with open-vocabulary, or more readily extensible alternatives, would strengthen the evaluations", "was_revised": true, "final_justification": "The authors have addressed my concerns regarding \n1. Providing a comparison to non-interpretable baselines\n2. Potential inflexibility of the framework by providing a an example of how one would avoid full retraining \nand have clarified a point of confusion. \n\nThe paper presents a well-motivated, high-performance method for interpretable video action recognition. The paper is lucid and, with the additional discussions/experiments, the well-rounded. Therefore, I maintain my recommendation of 5 for acceptance.", "rebuttal": "We thank the reviewer for the helpful comment.\n### **Clarification on the baseline in Table 1**\nWe apologize for the lack of clarity regarding the \"baseline w/o interpretability\" and other baselines in Table 1. The \"baseline w/o interpretability\" refers to a standard video recognition model trained without the linear concept layer. Specifically, we fine-tune a video encoder, e.g., VideoMAE, and a linear classification head directly on each target dataset without any concept supervision. We use Somethine-Something-v2-pretrained VideoMAE weights for KTH, Penn Action, and HAA-100, and UCF101-pretrained weights for UCF-101 to match the pretraining setup used in DANCE. Since DANCE adopts VideoMAE as its video backbone fine-tuned on each target dataset, we report the Top-1 accuracy of VideoMAE across datasets as the “baseline w/o interpretability” in Table 1. Also, for a fair comparison, isolating the effect of interpretability, we use the same feature extractor (VideoMAE) across all baselines (e.g., CBM [1] with UCF-101 attributes, LF-CBM [2] with entangled language concepts, and LF-CBM [2] with disentangled language concepts). This follows the common practice in ante-hoc explainable models [1,2,5,6], where the baseline is typically defined as the same backbone architecture trained for the target task without the interpretability. While VideoMAE is used throughout Table 1 of the main paper for consistency, the backbone is interchangeable, and we additionally report results with other backbones (e.g., CNN) in Table 10 of the supplementary material.\n\n### **Justification for SOTA Comparison**\nWe agree that the performance of DANCE is lower than recent SOTA results on UCF-101 (e.g., 99.7%). However, our goal is not to compete with SOTA in recognition accuracy. Instead, our focus lies in injecting interpretability into video backbones and analyzing the trade-off between interpretability and performance in Table 1 of the main paper.\nNevertheless, to examine the scalability of DANCE and its compatibility with stronger architectures, we apply it to AIM [3] and InternVideo2 [4]—both of which are more powerful backbones than VideoMAE.  Specifically, AIM is fine-tuned on UCF-101 (AIM is a parameter-efficient fine-tuning method using a pre-trained CLIP as its backbone), while InternVideo2 is used directly with its IV-400M pretraining.  We present the results in the table below.\n| Method         | Interpretability | Top-1 Accuracy |\n|----------------|:------------------:|:-----------------------------------:|\n| AIM           | ✗                | 94.5                       |\n| DANCE with AIM backbone         | ✓                 | 93.6                       |\n| Internvideo2 ( visual )    | ✗                | 97.3                       |\n| DANCE with InternVideo2 backbone ( visual )      | ✓                | 96.5                       |\n\nAIM alone achieves 94.5% on UCF-101, and DANCE with AIM backbone achieves 93.6%. Similarly, InternVideo2 achieves 97.3% alone, and DANCE with InternVideo2 backbone achieves 96.5%. Specifically, we use only the frozen visual encoder of InternVideo2 as our video feature extractor. These results show that DANCE consistently benefits from stronger backbones while maintaining competitive performance.\n\n### **More context on DANCE’s performance relative to baselines**\nWe provide additional context on how DANCE performs relative to baselines. Because DANCE enforces the video encoder to predict concepts through a linear concept layer, there might be a trade-off between accuracy and interpretability [1]. In this setting, it becomes crucial to assess how well each method maintains recognition performance while providing meaningful explanations. As shown in Table 1 of the main paper, DANCE achieves higher accuracy than other concept-based methods on KTH and Penn Action, while showing only a modest drop of 2.8 points on HAA-100 and 0.9 points on UCF-101 compared to the baseline w/o interpretability. Furthermore, as shown in Figure 6 of the main paper, DANCE receives the highest user ratings in the user study evaluating explanation quality. These results demonstrate that DANCE achieves the best trade-off between interpretability and performance, validating the effectiveness of our explanation design.\n\n\n### **Flexibility-interpretability trade off**\nWe acknowledge that DANCE prioritizes interpretability, which may impose some constraints on flexibility. However, adding a new class does not require redoing concept discovery from scratch.\n\nFor object and scene concepts, we can extract concepts from the new class and apply concept pseudo-labeling for the new concepts and the samples belonging to the new class (see Eq. (3) in the main paper). For motion dynamics concepts, we can discover new clusters from the new class’s pose sequences using a semi-supervised clustering approach [7], allowing us to incrementally expand the concept space when novel motion patterns emerge. Therefore, we can add a new class by updating concept labels and retraining only the concept layers and the linear classifier. Retraining takes only $\\sim 8$ minutes with a single RTX 3090 GPU on the UCF-101 dataset.\n\nTo empirically verify this, we simulate a class addition scenario on the UCF-101 dataset by randomly holding out one class during training. After training DANCE on the 100 classes, we update the object, scene, and motion dynamics concept labels for the held-out class without concept discovery from scratch. We show the results of this scenario in the table below.\n\n| Method| Top-1 Accuracy |\n|---------------------|:---------:|\n| DANCE before adding a new class |   87.2    |\n| DANCE after adding a new class  |   86.8    |\n\nThe marginal accuracy drop (< 0.5 points) confirms that DANCE supports efficient class extension without re-running the full concept discovery pipeline. While fully incremental learning (i.e., class-incremental learning with many classes) is outside the scope of this work, we believe this experiment demonstrates that DANCE is more extensible than a rigid, from-scratch formulation.\n\n### **Comparison with open-vocabulary methods**\nTo address the reviewer’s concern, we compare the performance of DANCE with recent open-vocabulary models on the UCF-101 dataset. We present the results in the table below.\n| Method   | Total Param.     | Interpretability | Top-1 Accuracy |\n|------------|:----:|:------------------:|:----------------------------:|\n| CLIP [8]          |150M| ✗                | 68.9                       |\n| TVTSv2 [9]      | 956M | ✗                | 78.0                       |\n| InternVideo2 [4]| 6B | ✗                | 89.5                       |\n| **DANCE**    | 90M  | ✓                | 87.5                       |\n\nWhile InternVideo2 shows the highest accuracy (89.5%), it is a 6B-scale foundation model with multi-modal capabilities. These results suggest that DANCE achieves a favorable balance between interpretability and performance, remaining competitive even when compared to large-scale open-vocabulary methods.\n\n### **Clarification on concept deactivation**\nWe apologize for the lack of clarity. In this context, “deactivating” a concept means setting its activation to zero, not its weight. This allows us to assess the influence of each concept by directly manipulating the output of the concept layer while keeping the learned weights unchanged. We will clarify this point explicitly in the final version.\n\n\n##### **References**\n[1] Koh, Pang Wei, et al. \"Concept bottleneck models.\" ICML 2020.\n\n[2] Oikarinen, Tuomas, et al. \"Label-free Concept Bottleneck Models.\" ICLR 2023.\n\n[3] Yang, Taojiannan, et al. \"AIM: Adapting Image Models for Efficient Video Action Recognition.\" ICLR 2023.\n\n[4] Wang, Yi, et al. \"Internvideo2: Scaling foundation models for multimodal video understanding.\" ECCV 2024.\n\n[5] Yuksekgonul, Mert, Maggie Wang, and James Zou. \"Post-hoc Concept Bottleneck Models.\" ICLR 2023.\n\n[6] Shang, Chenming, et al. \"Incremental residual concept bottleneck models.\" CVPR 2024.\n\n[7] Vaze, Sagar, et al. \"Generalized category discovery.\" CVPR. 2022.\n\n[8] Radford, Alec, et al. \"Learning transferable visual models from natural language supervision.\" ICML 2021.\n\n[9] Zeng, Ziyun, et al. \"Tvtsv2: Learning out-of-the-box spatiotemporal visual representations at scale.\" arXiv preprint arXiv:2305.14173 (2023)."}, {"review_id": "OF2IAxe0Mt", "reviewer": "Reviewer_DRfP", "summary": "DANCE, the method, inputs a video and aims to explainably classify the action occuring in said video. Each sample video during training also includes a binary presence/absence vector for concepts. \n\nA pre-trained encoder is used to extract features from the video. These features are projected with a linear concept layer into M activations (where M is the number of concepts), with each projection corresponding to a concept. This concept projection layer has weights that correspond to different types of concepts, specifically motion dynamics, objects, and scene concepts. The goal of this process is an explainable \"concept activation\" vector of size M. This vector is apportioned into a section related to motion dynamics, objects, and scenes. This vector is then used to predict a final action.\n\nPrior to training this method, concept discovery is done. To get motion related concepts, the authors use key clip selection to find chunks with motion cues. They estimate 2D pose and for each frame in a clip of length L and then cluster the motion patterns. After doing this across the training dataset, they can extract clusters as motion concepts. For the concept activation vector with activations related to motion, each index relates to these clustered motion concepts. For each training video, that video/clip has that motion concept annotated if it belongs to that cluster. \n\nBeyond motion concepts, object and scene concepts are generated for action classes by using VLMs. Specifically, an LLM first pseudo-annotates what objects are related to an action class, and then vision-language dual encoder is used to determine if those objects are present in the clip. Similarly to motion concepts, the clip is then annotated with the presence/absence of these objects through a similarity score between features. \n\nOverall, the paper finds ways to pseudo-annotate motion, object, and scene concepts in video clips. These pseudo-annotations come from discovered concepts in both the data and how the data relates to an LLM's understanding of what the actions should involve. After concept discovery, the presence of these concepts is represented as part of the sample.\n\nNext, training is done to predict the presence/absence of concepts independently of the action prediction. After this training is complete, the concept layer is frozen and the final classifier is trained. This separation of concerns leads to an interesting and interpretable way of understanding how concept activations lead to action classifications, as the concept scores are introspectable.", "questions": "LLMs / VLMs were used for data annotation, couldn't they have been used for an additional baseline? It would be nice to see performance of more modern / more broad methods. \n\nAlthough language is a poor description of motion as shown in Figure 1, I'm curious how a method that was solely based on language would perform. Other prior work on relationship detection and other related action topics have shown strong priors based on the presence/absence of objects and/or just simply using language models on textual descriptions of the videos.", "limitations": "Motion clustering has implications for surveillance as it provides an easy data-centric way to extract concerning human poses. This type of easy-to-detect pose sequence clustering could have negative impacts as it is a lot easier to run this clustering on the outputs of a 2D pose detector over time than it is to process all the video in other more computationally expensive ways.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "This paper follows a tried and true methodology of data-centric concept discovery leveraged in the training of a classifier for use in a downstream task. The ability to pseudo-label annotate the concepts of the training data in order to first train a concept predictor and then action predictor is useful for creating an interpretable model. \n\nThe user study shows good performance for this method, and it is also useful quantitatively as it performs on par with the baseline w/o interpretability. \n\nRich interpretability as proposed in DANCE is useful both for better introspection into why/how classifiers make decisions and when a data-centric concept-learning phase as used in DANCE can actually help outperform uninterpretable baselines by grounding the focus of the classifier on certain relevant/salient distinctions (such as motion categories). Overall, this paper is quite interesting and helps expand the XAI domain. \n\nWeaknesses include the limitation of the motion-related concepts to human pose. Ideally, motion concepts could be more general and not just human-centric (although this is being used for action recognition).", "was_revised": true, "final_justification": "The authors adroitly responded to my review. The paper is interesting and the other reviewers agreed on our ratings.", "rebuttal": "### **Toward generalization beyond human-centric motion dynamics concept**\n\nWe sincerely thank the reviewer for highlighting the limitation regarding the scope of our motion dynamics concepts. We understand and appreciate the concern that relying solely on human pose may restrict the generality of motion representation. \nIn this work, we focus on explaining *human actions* in videos, where pose sequences provide structured and semantically meaningful cues for motion understanding.\n\nGiven our focus, we adopt human pose sequences to capture *fine-grained temporal patterns*. This representation enables users to intuitively understand how an action unfolds over time without being distracted by irrelevant visual factors such as clothing or background. Importantly, we position this work as an initial step toward ante-hoc explainable video understanding. Our goal is to establish a clean and modular framework grounded in disentangled concept types—motion dynamics, object, and scene—that allows structured, faithful, and intuitive explanation. Starting with human pose is a practical and task-aligned choice that provides a solid foundation for further development.\n\n#### **Limitations in non-human-centric videos**\nIn scenarios where humans are not present, DANCE can still produce explanations based on object and scene concepts. While these allow for reasonable interpretation of the spatial context, they may be less effective at capturing fine-grained object motions or physical interactions. We acknowledge this limitation and further discuss it in the limitations and broader impact section of the supplementary materials.\n\n#### **Future directions for generalizing motion dynamics concept**\nTherefore, we agree with the reviewer that extending motion dynamics concepts beyond human pose is an important and exciting direction. We plan to explore non-human-centric motion representations such as flow-based dynamics and object trajectories.\nConcretely, one possible approach is to leverage a video segmentation model such as Sam2 [1] *to capture the movement of salient objects* over time and define such patterns as motion concepts. These non-human-centric motion dynamics can be incorporated alongside our motion dynamics concept to create a more structured and general representation of motion. This unified formulation would allow DANCE to reason about both human actions and object-level dynamics in a coherent and interpretable manner.\nWe hope that this work encourages further exploration in the underexplored area of ante-hoc explainable video models.\n\n### **Additional baseline**\n\nWe appreciate the reviewer’s suggestion to consider more modern and broad models such as vision-language models (VLMs) as baselines. To address this, we provide a comparison with InternVideo2 [2], a large-scale VLM trained on extensive video-text pairs, which supports both zero-shot inference via text generation and classification via linear-probing.\n\n| Method|Interpretability |Top-1 Accuracy|\n|--|:-:|:-:|\n| InternVideo2 zeroshot  |× |89.5|\n| InternVideo2 visual encoder|× |97.3|\n|  DANCE with InternVideo2 visual encoder |✓|  96.5| \n\nAs shown above, InternVideo2 achieves strong performance (89.5% zero-shot ). While such models can produce textual rationales via prompting (e.g., chain-of-thought), these outputs are not faithful to the model’s internal reasoning and are hard to validate [3-5]. Additionally, current VLMs still lack fine-grained temporal understanding, limiting their suitability for structured, explanation-driven action recognition (see the last two paragraphs of Section 6.2 in the supplementary).\nTherefore, rather than using VLMs as standalone explainable models, we take a more practical and structured approach by employing their strong visual encoders within the DANCE framework, which is inherently interpretable. Specifically, we employ  InternVideo2’s visual encoder as the feature extractor $f(\\cdot)$ in DANCE while keeping the rest of the pipeline unchanged. This variant achieves 96.5% accuracy—comparable to the original InternVideo2—while offering concept-based interpretability. This result demonstrates the scalability of our framework and its ability to integrate strong backbones like InternVideo2 without sacrificing explainability.\n\n### **Comparison with \"language-based explanation baselines\"**\n\nIn this work, we evaluate language-based explanation baselines, as shown in Figure 1 (b), Table 1, Figure 6, and the user study section (L258-282) of the main paper. Specifically, the method in Figure 1 (b) follows the Label-Free CBM (LF-CBM) [9] pipeline, where spatial and temporal concepts are derived directly via prompt engineering from GPT-4o, without manual annotations.\n\nTo assess their effectiveness, we conduct a user study comparing explanations from LF-CBM with those of our motion dynamics concepts (Figure 6 of the main paper). For a fair comparison, both methods use the same video backbone (VideoMAE), ensuring that differences arise solely from the concept extraction process. In summary, our user study shows that motion dynamics concepts offer more intuitive and structured explanations than language-based ones, achieving higher absolute ratings (**Ours (4.3) vs Language-based (2.3) in Figure 6 (b)**) and **75% preference** relative to \"GPT\" (language-based baseline) in Figure 6 (a).\n\nImplementation details and evaluation protocols are provided in Sections 4.2 and 5.1 of the supplementary material. Additional qualitative examples of language concept-based explanations and discussions are in Figure 9, Figure 10, and L616-646 of the supplementary material. We hope this clarifies how language-only concept supervision is incorporated and evaluated in our framework. \n\n\n### **Language-based and relationship detection approaches**\n\nWhile these methods show promising performance by aligning visual content with textual semantics, DANCE differs from them in key aspects.\n\n**(1) Relationship detection methods** typically require fine-grained relational labels (e.g., subject–object–action triplets) and rely on vision-language grounding to detect human-object interactions (HOIs) [6–8] . However, such annotations are not available in general action recognition datasets like UCF-101 or Penn Action, limiting their scalability without additional manual labeling. Moreover, most of these methods do not explicitly aim to provide interpretability. \n\n**(2) Language-only video descriptions** (e.g., using captioning models or VLMs) can generate high-level textual summaries, but they still operate in a black-box manner and do not provide faithful explanations of why a specific prediction was made [3-5]. Their descriptions serve as task outputs—summarizing content or enabling retrieval—rather than providing transparent access to the model’s internal decision-making process. As such, they belong to a different category of research, focused more on generation or alignment than explanation. In contrast, our work focuses on making the model’s reasoning process explicit and structured through disentangled concept representations, aligning more closely with the goal of faithful and ante-hoc interpretability.\n\n### **Goal of our work**\nRather than comparing DANCE explanations with the final outputs of language-based models, we compare them with explanations from language-based Concept Bottleneck Models (CBMs), which aligns better with our goal of making the model’s internal reasoning transparent.\n\nWhile recent works [9–11] in the image domain explore CBMs using language-derived concepts (e.g., CLIP embeddings), we argue that such representations are limited in the video domain due to the lack of temporal modeling and implicit motion cues. To address this, we propose motion dynamics concepts grounded in pose-based temporal patterns, complementing object and scene concepts from vision-language models. This structured, disentangled design enables DANCE to produce faithful, motion-aware explanations tailored for video understanding.\n\n### **Concern about surveillance**\nWe acknowledge the potential for misuse. However, our method builds on standard pose estimation and unsupervised clustering—tools already widely available—and does not infer intent or sensitive attributes. Its design centers on interpretability, not surveillance. We will include a disclaimer on responsible use in the broader impact section.\n\n\n\n#### **References**\n[1] Ravi, Nikhila, et al. \"SAM 2: Segment Anything in Images and Videos.\" ICLR, 2025.\n\n[2] Wang, Yi, et al. \"Internvideo2: Scaling foundation models for multimodal video understanding.\" ECCV 2024\n\n[3] Wang, Boshi, et al. \"Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matters.\" ACL 2023.\n\n[4] Arcuschin, Iván, et al. \"Chain-of-Thought Reasoning in the Wild is not Always Faithful.\" Reasoning and Planning for LLMs Workshop, ICLR 2025.\n\n[5] Turpin, Miles, et al. \"Language models don't always say what they think: Unfaithful explanations in chain-of-thought prompting.\" NeurIPS 2023.\n\n[6] Wang, Ning, et al. \"Language model guided interpretable video action reasoning.\" CVPR 2024.\n\n[7] Zhao, Long, et al. \"Unified visual relationship detection with vision and language models.\" ICCV 2023.\n\n[8] Wang, Yongqi, et al. \"End-to-end open-vocabulary video visual relationship detection using multi-modal prompting.\" TPAMI 2025.\n\n[9] Oikarinen, Tuomas, et al. \"Label-free Concept Bottleneck Models.\" ICLR 2023.\n\n[10] Yuksekgonul, Mert, Maggie Wang, and James Zou. \"Post-hoc Concept Bottleneck Models.\" ICLR 2023.\n\n[11] Shang, Chenming, et al. \"Incremental residual concept bottleneck models.\" CVPR 2024."}, {"review_id": "D3tiLZExX0", "reviewer": "Reviewer_JS41", "summary": "The paper proposes DANCE, an ante-hoc, concept-bottleneck framework for video action recognition that forces the model to reason through three disentangled and human-interpretable concept spaces: motion dynamics, objects, and scenes. Motion concepts are created without labels by clustering 2-D pose sequences extracted from automatically detected key clips, giving appearance-invariant representations of how the body moves. Object and scene concepts are obtained by querying GPT-4o for class-specific keywords and then pseudo-labeled in videos via a vision–language dual encoder.", "questions": "- How do you use GPT-4o for videos, I thought it does not allow videos. Do you pass in key image frames and the model handles it as separate images?\n- LLM-generated lists are ad-hoc, prompt-dependent, and not guaranteed to be exhaustive. Rare but crucial objects may be absent. How do you handle if your test videos has something unseen yet?\n- Forcing predictions through a relatively small concept basis would hurt recognition, for instance if a new shape appears, the model must either conflate it with an existing object concept or ignore it. Does accuracy degrade on rare subclasses?\n- For the CLIP thresholding you mention, how is thresholding of CLIP similarity performed to binarise object/scene labels? Is it class-agnostic or tuned per concept?\n- How many pose sequences and how many FINCH clusters per dataset? It would be very useful to analyze this since we probably do not want a lot of clusters.", "limitations": "- Gains of +1% on small datasets (like shown) can easily fall within variance from random initialisation or sampling. No standard deviation across seeds is reported.\n- One big limitations is all the clusters are precomputed and it is very hard to generalize outside what you have already seend uring making the clusters.\n- How does DANCE handle actions that involve two coordinated agents (e.g., handshake) or tool–object interaction where the object itself moves (e.g., “pull-ups on bar”)? The current motion representation is strictly single human-centric.\n- Pose snippets are fixed-length (L frames). Actions whose discriminative cue lasts longer than L are split across several clusters and the bottleneck has no mechanism to model their ordering.\n- Skeleton co-ordinates are pixel-based, not camera-normalised. Without 2-D/3-D normalisation, clusters entangle action with camera geometry.\n- Motion clusters and GPT lists are dataset‐specific.Porting the model to a new domain (e.g., underwater sports apart from just sports like the examples) requires re-running both pipelines and retraining all heads which make it hard to follow the argument that “concepts give modularity”. The framework also does not work for anything different than one human doing some task in a exocentric view.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 2, "strengths_and_weaknesses": "## Strengths\n\n- The paper tackles the under-explored but important question of how to produce human–aligned explanations for video action recognition, explicitly separating the “how” (motion) from the “what/where” (object- and scene-level context).\n- By making the solution as an ante-hoc concept bottleneck with three disjoint partitions, the authors provide a simple yet expressive inductive bias that is easy to reproduce and to reason about.\n- Object and scene concepts are sourced automatically with a single prompt to an LLM, eliminating any handrcrafting which you often see in many concept-based models.\n- The pipeline is modular: each component (pose estimator, clustering algorithm, vision–language encoder) can be swapped for upgrades without retraining the full network.\n\n## Weaknesses\n\n- The novelty is limited. The architecture is a textbook concept-bottleneck pipeline (linear projection -> sparse linear head) is quite standard. Also disentangling the bottleneck into three blocks does not guarantee disentanglement in the learned representation. The main changes the authors make are mainly highly handcrafted ways (this causes man of the other limitations and weaknesses) of disentangling into 3 components.\n- Two-stage training (first concepts, then classifier) does not preclude “shortcut” interactions between concepts at evaluation time. A stronger test would be to interchange concept vectors between unrelated videos and show that predictions change in an intuitive way.\n- Clustering 2-D poses from short “key clips” is heuristic. Clusters might mix mutually incompatible motion primitives (e.g., mirrored left/right variants) or fragment a single semantic motion across multiple clusters. Something like mutual information with ground-truth actions, intra-cluster variance, or human-label agreement is needed to verify that the clusters are indeed coherent though the results shown actually look good.\n- Similar to playing a clip backwards which is a decent diagnostic, but the paper does not try more extrem tests like re-colouring frames, jittering the bounding box, or removing the person entirely would test whether the model truly disentangles motion from context.\n- FINCH requires pairwise nearest-neighbour computations over all pose sequences. For Something-Something or Kinetics this will be really large, is that why you only tests on smaller datasets?\n- Nothing prevents the model from silently exploiting residual information in the frozen backbone that never surfaces in the linear concept layer. SImilarly, when parent concepts are correlated (e.g., “baseball-bat” always co-occurs with “swing-pose”), gradients can flow through a single node and the classifier still reaches high accuracy.\n- The linear concept layer is extremely overcompletefd (hundreds of motion clusters + hundreds of GPT objects/scenes) but the only sparsity pressure is a l1 term on the classifier, not on the concept activations themselves. Activations therefore remain dense, defeating the “bottleneck” intuition and hampering faithfulness.", "was_revised": true, "final_justification": "I think quite a few of my questions are addressed, some limitations like what clusters are used and the method only working for 1 human are still there like i indicated earlier. i had anyways given a high score and so i stick to it.", "rebuttal": "### **Clustering**\n#### **Key clips**: We determine $L$ for each dataset based on the overall video lengths and validation performance, as detailed in Supp. Table 4. To better capture discriminative cues in longer actions, we plan to incorporate temporal action localization methods that propose informative segments. Given a proposal, we can uniformly sample $L$ frames within the localized region, ensuring that key motion information is preserved without increasing model complexity.\n\n#### **Fragmented motions**: Regarding fragmented motions across multiple clusters, we clarify that this is a deliberate design. Actions like a \"long jump\" involve distinct motion phases (e.g., running and jumping), which we capture as separate motion dynamics concepts. This modularity supports compositional generalization and encourages concept reuse across different actions, e.g., reusing a running concept for \"high jump\". \n\n#### **Mirrored motions**: Through manual inspection, we find some clusters contain mirrored variants (e.g., left- and right-handed motions). To address this, we increase the number of pose clusters from 0.5K to 2K on UCF-101, and we observe that left/right-specific clusters naturally emerged—suggesting that clustering granularity can help capture such variations when needed.\n\n#### **Coherence**: On Penn Action, we cluster 14K pose sequences into 80 groups using FINCH and evaluate concept coherence via NMI with action labels—0.62—indicating strong semantic alignment. Intra-cluster variance (12.1±5.5) is much lower than global variance (66.2), showing tight grouping. A human study with 50 participants yields a 4.2 ± 0.7 Likert score on visual similarity, confirming high perceived coherence.\n\n#### **Scalability**: FINCH takes 49.3 seconds on UCF-101 (10K sequences) and 200.9 seconds on Kinetics-400 (240K sequences), which remains efficient and does not significantly impact the overall pipeline. This is because the original FINCH paper provides an approximate nearest neighbor (ANN) method, reducing the complexity from $O(N^2)$ to $O(N log N)$. While we can run FINCH clustering on Kinetics-400 or Something-Something-V2, we do not run our model on these datasets because of data-quality considerations, e.g., humans not visible, unreliable pose estimation due to occlusion. We plan to extend our work to more general scenarios in future work. \n\n#### **Precomputed clusters**: Our motion dynamics concepts capture reusable primitives across classes—for example, “down-swing” appears in both *badminton overswing* and *volleyball overhand* (Supp. Fig. 13). To assess generalization, we simulate a class addition scenario on UCF-101 by holding out one class during training. We then assign its pose sequences to existing clusters without retraining. DANCE achieves 87.1% accuracy after adding the class, compared to 87.2% before—a negligible drop. This suggests that the clusters generalize well to unseen classes.\n\n### **Shortcut**\nTo directly test for shortcut interactions, we perform concept interchange experiments on Penn Action.\n\n**Scenario I: Similar contexts, different motions**: We swap motion dynamics between *Tennis forehand* and *Tennis serve*, which share scene/object context. The predicted classes also swap with high confidence, demonstrating that DANCE bases its decision on motion concepts rather than static context, confirming their disentangled influence.\n\n||**Motion dynamics swap in a *tennis forehand* video**||\n|-|:-:|:-:|\n|| Tennis forehand conf.| Tennis serve conf.| \n|Before| 0.98|0.01|\n|After|0.01|0.98|\n\n|| **Motion dynamics swap in a *tennis serve* video**||\n|-|:-:|:-:|\n|| Tennis forehand conf.| Tennis serve conf.| \n|Before| 0.01|0.99|\n|After|0.98|0.01|\n\n**Scenario II: Unrelated actions**: We swap scene concepts between *Golf swing* and *Bench press* videos. Golf swing’s confidence drops notably due to its unique spatial context within the Penn Action dataset, while Bench press is unaffected as it shares context with other gym actions where motion dominates (e.g., Squat and Clean & Jerk), within the dataset. This intuitive response to concept swaps shows that DANCE does not exploit shortcuts and that its predictions are grounded in the appropriate concept type.\n\n||**Scene swap in a *golf swing* video**||\n|--|:-:|:-:|\n|| Golf swing conf.  | Bench press conf.| \n|Before| 0.98|0.01|\n|After|0.54|0.01|\n\n|| **Scene swap in a *bench press* video**||\n|--|:-:|:-:|\n|| Golf swing conf.  | Bench press conf.| \n|Before| 0.00|0.99|\n|After|0.00|0.99|\n\n### **Concept disentanglement**\nIn Supp. Section 6.2 and Figure 16, we create static videos by replicating a single frame across time. For example, replicating a handstand frame from a balance beam video leads DANCE to predict HandstandWalking—a class with a similar pose but with a different context. The result suggests that DANCE successfully disentangles motion from spatial context and makes predictions based on the underlying motion dynamics.\n\nIn addition, we apply color jittering to every frame of a normal video during inference on UCF-101. Specifically, we randomly perturb brightness (±40%), contrast (±40%), saturation (±40%), and hue (±20%) for each frame. The overall top-1 accuracy slightly drops from 87.5% to 86.3%, indicating that DANCE remains robust even under significant appearance distortions. \n\n### **Residual information**\nWe mitigate leakage by freezing the backbone and enforcing a strict bottleneck: predictions are made only from a sparse linear layer. While some concept correlations (e.g., “baseball-bat” and “swing-pose”) may persist, our framework enables *explicit inspection* and *intervention*, improving transparency despite imperfect disentanglement.\n\nAs shown in Fig. 10 (UCF101-SCUBA), both VideoMAE and DANCE initially perform poorly (<4%) due to context bias. However, manual edits to DANCE recover 72 samples, raising accuracy to 35.0%, while VideoMAE remains unchanged. This demonstrates DANCE’s practical advantage: user-guided correction enabled by transparency, even when a model inherits bias from the frozen encoder.\n\n### **Sparsity**\nWe observe that classifier sparsity remains below 5% across all experiments, indicating reliance on a small set of meaningful concepts. At the sample-level explanation (e.g., Fig. 11, “punching sandbag” Supp.), the top-5 concepts contribute 9.95 to the logit, while the remaining 495 contribute only 0.8—under 8%. This confirms that predictions are driven by a few essential concepts, supporting faithfulness.\n\n### **Novelty**\nWhile our framework adopts the standard CBM architecture, our key novelty lies in extending CBMs to the *video domain* through a *structured, interpretable decomposition* into motion dynamics, object, and scene concepts. This tripartite disentanglement is non-trivial in spatio-temporal settings and well aligned with human cognition. Furthermore, our approach uses unsupervised motion concept discovery and zero-shot LLM supervision for object and scene concepts, which together enable interpretable representations without manual annotation or task-specific and heavy engineering.\n\n### **GPT-4o for videos**\nWe input class names as text prompts, not images or videos, to generate related object and scene concepts. See Section 1.3.2 and 4.2 of Supp. for more details.\n\n### **Rare objects**\nWhile LLM-generated concepts may miss rare objects, GPT-4o yields robust and representative concepts from action names alone, outperforming LLaVA, which uses key frames for concept extraction (84.6% vs. 82.9%; see Supp. Table 5, Sec. 6.1.1).\nMoreover, DANCE’s modular design allows it to fall back on available concept types. As shown in Fig. 10, it remains effective under domain shifts by relying solely on motion dynamics. Even when all concepts fail, DANCE still supports interpretable failure analysis—unlike black-box models.\n\n### **Small concept basis**\nWe acknowledge that a small concept basis may impact recognition. However, as shown in Supp. Table 6, increasing motion concepts yields only marginal gains, suggesting sufficiency. To test robustness on rare subclasses, we group UCF-101 into head/mid/tail classes. DANCE achieves 87.5% overall, with 87.9%, 85.2%, and 89.5% on head, mid, and tail groups—demonstrating strong performance even on underrepresented classes.\n\n### **CLIP thresholding**\nWe do not binarize object or scene concept labels but treat them as soft labels ranging from 0 to 1. Instead, we apply fixed threshold-based filtering at two stages: (1) initial activation filtering that removes concepts with low average ViCLIP activations and (2) projection filtering that removes low projected concept activations after optimization. This class-agnostic strategy follows LF-CBM and involves no per-concept or per-class tuning. See Supp. Sections 1.3.2 and 2.2 for details.\n\n### **Pose sequences**\nFINCH yields a hierarchy of motion concepts at varying granularities. For UCF-101, ~100K pose sequences produce six levels, from 22,706 fine-grained to 10 coarse clusters. This allows flexible abstraction levels when selecting motion concepts.\n\n### **STD**\nWe report the mean and standard deviation across 5 random seeds as follows.\n\n|KTH|Penn Action|HAA-100|UCF-101|\n|:-:|-:|-:|-:|\n|91.18±0.02|98.08±0.00|70.76±0.65|87.58±0.10|\n\n### **Skeleton**\nAs described in Section 1.3.1 Supp., we normalize each pose by centering the joint coordinates and scaling them to the range [0, 1].\n\n### **Two agents**\nWe focus on single-human actions as a first step toward explainable video recognition. While our current framework does not yet handle multi-agent or object-centric motions, future work will explore multi-person inputs, relation-aware clustering, and segment-based trajectory encoding (e.g., SAM2 [Ravi et al, ICLR 2025] ) to generalize motion dynamics beyond single-human scenarios.\n\n### **New class/domain**\nPlease refer to the fourth paragraph of our response to Reviewer nxgA: Flexibility-interpretability trade off."}, {"review_id": "bHd4k06BKW", "reviewer": "Reviewer_g8Sg", "summary": "The authors propose the Disentangled Action aNd Context concept-based Explainable (DANCE) framework.\nThe goal is to provide concept-based explanations, which can belong to one of three categories: motion dynamics, objects, or scenes. Motion dynamics concepts are automatically discovered by clustering pose sequences. They assess the effectiveness of their method through experiments conducted on four distinct datasets.", "questions": "1. Do you have results for DANCE model performance when excluding a specific type of concep, such as the motion dynamics category, to demonstrate its necessity?\n\n2. You mention a baseline model (lines 295–296), but I am not entirely sure what this model entails or how it differs from the model proposed in the DANCE framework. Could you please clarify?\n\n3. What was the motivation for adopting a two-stage training process, as described on lines 210–211?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths :**\nThe method section is clear, with a key contribution being the introduction of the motion dynamic concept category and demonstrating how these concepts can be discovered within the model without relying on human annotation. Additionally, a notable strength is the inclusion of a user study to evaluate the quality of the explanations in comparison to other methods.\n\n**Weaknesses:** It seems that the main contribution lies in the disentangled explanations and the introduction of the motion dynamics concept category. Therefore, I would have expected to see results demonstrating the benefit of incorporating these additional concepts (see Question 1).\nIn Table 1, I am not sure I fully understand what the baseline model represents (see Question 2).", "was_revised": true, "final_justification": "The authors have addressed all of my questions, and as a result, I have increased my rating.", "rebuttal": "### **Contribution of Disentangled Concept Types**\n\nWe thank the reviewer for the insightful question regarding the contribution of each concept type. We agree that analyzing the effect of each concept type is important, and we have already conducted such an ablation study on the UCF-101 dataset. We reported in Table 9 of the supplementary material. For clarity, we summarize the key findings below.\n\nUsing only **motion dynamics concepts** achieves **86.4%** accuracy, demonstrating their strong standalone importance. **Adding object or scene concepts** on top of motion dynamics increases accuracy to 86.7% and 86.5%, respectively—indicating that contextual information provides complementary cues, with object concepts offering slightly more action-relevant signals than scene context. In contrast, using **only object and scene concepts** without motion dynamics results in a notable performance drop to **84.6%**, highlighting the limitations of relying solely on spatial context for recognizing video actions.\nFinally, combining all three concept types—motion dynamics, object, and scene—achieves the highest accuracy of **87.5%**, confirming the complementary nature of **disentangled spatio-temporal concepts**.\n\nThese results validate the importance of motion dynamics and demonstrate the effectiveness of our structured concept representation in improving both interpretability and classification performance.\n\n### **Clarification on the Baseline Model in Table 1**\n\nWe apologize for the lack of clarity regarding the baseline model reported in Table 1 of the main paper. The entry *Baseline w/o interpretability* refers to a standard video recognition model that does not incorporate any concept supervision. Specifically, we fine-tune a video encoder  (VideoMAE) and a classification head directly on each target dataset **without any concept supervision**. For fair comparison, we use Something-Something-v2-pretrained VideoMAE weights for KTH, Penn Action, and HAA-100, and UCF101-pretrained weights for UCF-101, using the same pretrained weights as DANCE. We note that the VideoMAE used in *Baseline w/o interpretability* is identical to the video feature extractor employed in DANCE, ensuring a fair backbone comparison.\n\n\nThe key difference from DANCE lies in **the absence of the concept layer**, which in our framework consists of three disentangled types: motion dynamics, object, and scene concepts. This baseline follows a common protocol in recent CBM-based approaches[1-4], where the backbone remains the same and only the concept layer is removed.\n\nTherefore, this non-interpretable setup serves as a reference point to assess **the trade-off between accuracy and interpretability** introduced by DANCE. We will clarify this setup explicitly in the final version of the paper.\n\n### **Motivation for adopting a two-stage training process**\n\nThank you for your question regarding our two-stage training strategy. As described in lines 210–211 of the main paper, we first train the concept prediction layers, and then train the linear classification layer while keeping all concept layers frozen. In general, there are **two possible training paradigms** in concept bottleneck models:\n\n(i) *sequential (two-stage)* — first train the concept layer, then the classification layer (our approach),\n\n(ii) *joint* — train both simultaneously using a total loss ( e.g., $L=L_{concept}+L_{classification}$ ).\n\nWe adopt the **two-stage** approach primarily due to our emphasis on explainability over pure classification performance. In joint training, the classification loss can influence the concept layer, potentially introducing shortcuts that prioritize end-task performance at the cost of faithful concept learning. By separating the training of the concept and classification layers, we encourage the concept layers to capture semantically **meaningful representations independent of the final labels**.\nThis motivation aligns with prior findings in the original CBM paper [1], which shows that sequential training better focuses on concept learning compared to joint training. Moreover, many recent CBM-based approaches adopt two-stage training for similar reasons [1-4].\n\n\n\n\n#### **References**\n[1] Koh, Pang Wei, et al. \"Concept bottleneck models.\" ICML 2020.\n\n[2] Oikarinen, Tuomas, et al. \"Label-free Concept Bottleneck Models.\" ICLR 2023.\n\n[3] Yuksekgonul, Mert, Maggie Wang, and James Zou. \"Post-hoc Concept Bottleneck Models.\" ICLR 2023.\n\n[4] Shang, Chenming, et al. \"Incremental residual concept bottleneck models.\" CVPR 2024."}]

Author rebuttal:

Meta-review: This work received mostly positive reviews in the initial review, with 2 borderline accept and 2 accept scores. While the work was generally seen as technically sound and timely, some concerns raised around the following key points: (i) ambiguous technical novelty that does not clarify how it moves beyond the standard concept‑bottleneck pipeline and modest performance gains, (ii) potential over-dependence on precomputed, dataset‑specific motion clusters and GPT‑generated object/scene lists can limit its generalization to new domains or unseen actions, and (iii) limited details on baselines and limited qualitative analysis such as robustness checks and cluster quality analysis to support interpretability and effectiveness claims.

The authors have provided a rebuttal that addresses many of these concerns effectively, as seen by the increase in scores to 3 accepts and 1 borderline accept. The AC agrees with the general assessment and recommends acceptance. It is a good paper that will generate a lot of discussion with the NeurIPS audience. The authors are strongly encouraged to incorporate the discussion from the rebuttal phase into the final, camera-ready version for completeness.
Decision: Accept (spotlight)