Title: Disentangled Concepts Speak Louder Than Words: Explainable Video Action Recognition
Venue: NeurIPS 2025
Accepted: yes
Keywords: Video Action Recognition

Explainable AI

XAI

Concept

Disentangling

Motion Dynamics

Context
Abstract: Effective explanations of video action recognition models should disentangle how movements unfold over time from the surrounding spatial context. However, existing methods—based on saliency—produce entangled explanations, making it unclear whether predictions rely on motion or spatial context. Language-based approaches offer structure but often fail to explain motions due to their tacit nature—intuitively understood but difficult to verbalize. To address these challenges, we propose Disentangled Action aNd Context concept-based Explainable (DANCE) video action recognition, a framework that predicts actions through disentangled concept types: motion dynamics, objects, and scenes. We define motion dynamics concepts as human pose sequences. We employ a large language model to automatically extract object and scene concepts. Built on an ante-hoc concept bottleneck design, DANCE enforces prediction through these concepts. Experiments on four datasets—KTH, Penn Action, HAA500, and UCF101—demonstrate that DANCE significantly improves explanation clarity with competitive performance. Through a user study, we validate the superior interpretability of DANCE. Experimental results also show that DANCE is beneficial for model debugging, editing, and failure analysis.


Peer reviews:
comments: SUMMARY:
The authors propose the Disentangled Action aNd Context concept-based Explainable (DANCE) framework.
The goal is to provide concept-based explanations, which can belong to one of three categories: motion dynamics, objects, or scenes. Motion dynamics concepts are automatically discovered by clustering pose sequences. They assess the effectiveness of their method through experiments conducted on four distinct datasets.

STRENGTHS AND WEAKNESSES:
**Strengths :**
The method section is clear, with a key contribution being the introduction of the motion dynamic concept category and demonstrating how these concepts can be discovered within the model without relying on human annotation. Additionally, a notable strength is the inclusion of a user study to evaluate the quality of the explanations in comparison to other methods.

**Weaknesses:** It seems that the main contribution lies in the disentangled explanations and the introduction of the motion dynamics concept category. Therefore, I would have expected to see results demonstrating the benefit of incorporating these additional concepts (see Question 1).
In Table 1, I am not sure I fully understand what the baseline model represents (see Question 2).

QUESTIONS:
1. Do you have results for DANCE model performance when excluding a specific type of concep, such as the motion dynamics category, to demonstrate its necessity?

2. You mention a baseline model (lines 295–296), but I am not entirely sure what this model entails or how it differs from the model proposed in the DANCE framework. Could you please clarify?

3. What was the motivation for adopting a two-stage training process, as described on lines 210–211?

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-24

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes DANCE, an ante-hoc, concept-bottleneck framework for video action recognition that forces the model to reason through three disentangled and human-interpretable concept spaces: motion dynamics, objects, and scenes. Motion concepts are created without labels by clustering 2-D pose sequences extracted from automatically detected key clips, giving appearance-invariant representations of how the body moves. Object and scene concepts are obtained by querying GPT-4o for class-specific keywords and then pseudo-labeled in videos via a vision–language dual encoder.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\n- The paper tackles the under-explored but important question of how to produce human–aligned explanations for video action recognition, explicitly separating the “how” (motion) from the “what/where” (object- and scene-level context).\n- By making the solution as an ante-hoc concept bottleneck with three disjoint partitions, the authors provide a simple yet expressive inductive bias that is easy to reproduce and to reason about.\n- Object and scene concepts are sourced automatically with a single prompt to an LLM, eliminating any handrcrafting which you often see in many concept-based models.\n- The pipeline is modular: each component (pose estimator, clustering algorithm, vision–language encoder) can be swapped for upgrades without retraining the full network.\n\n## Weaknesses\n\n- The novelty is limited. The architecture is a textbook concept-bottleneck pipeline (linear projection -> sparse linear head) is quite standard. Also disentangling the bottleneck into three blocks does not guarantee disentanglement in the learned representation. The main changes the authors make are mainly highly handcrafted ways (this causes man of the other limitations and weaknesses) of disentangling into 3 components.\n- Two-stage training (first concepts, then classifier) does not preclude “shortcut” interactions between concepts at evaluation time. A stronger test would be to interchange concept vectors between unrelated videos and show that predictions change in an intuitive way.\n- Clustering 2-D poses from short “key clips” is heuristic. Clusters might mix mutually incompatible motion primitives (e.g., mirrored left/right variants) or fragment a single semantic motion across multiple clusters. Something like mutual information with ground-truth actions, intra-cluster variance, or human-label agreement is needed to verify that the clusters are indeed coherent though the results shown actually look good.\n- Similar to playing a clip backwards which is a decent diagnostic, but the paper does not try more extrem tests like re-colouring frames, jittering the bounding box, or removing the person entirely would test whether the model truly disentangles motion from context.\n- FINCH requires pairwise nearest-neighbour computations over all pose sequences. For Something-Something or Kinetics this will be really large, is that why you only tests on smaller datasets?\n- Nothing prevents the model from silently exploiting residual information in the frozen backbone that never surfaces in the linear concept layer. SImilarly, when parent concepts are correlated (e.g., “baseball-bat” always co-occurs with “swing-pose”), gradients can flow through a single node and the classifier still reaches high accuracy.\n- The linear concept layer is extremely overcompletefd (hundreds of motion clusters + hundreds of GPT objects/scenes) but the only sparsity pressure is a l1 term on the classifier, not on the concept activations themselves. Activations therefore remain dense, defeating the “bottleneck” intuition and hampering faithfulness.\n\nQUESTIONS:\n- How do you use GPT-4o for videos, I thought it does not allow videos. Do you pass in key image frames and the model handles it as separate images?\n- LLM-generated lists are ad-hoc, prompt-dependent, and not guaranteed to be exhaustive. Rare but crucial objects may be absent. How do you handle if your test videos has something unseen yet?\n- Forcing predictions through a relatively small concept basis would hurt recognition, for instance if a new shape appears, the model must either conflate it with an existing object concept or ignore it. Does accuracy degrade on rare subclasses?\n- For the CLIP thresholding you mention, how is thresholding of CLIP similarity performed to binarise object/scene labels? Is it class-agnostic or tuned per concept?\n- How many pose sequences and how many FINCH clusters per dataset? It would be very useful to analyze this since we probably do not want a lot of clusters.\n\nLIMITATIONS:\n- Gains of +1% on small datasets (like shown) can easily fall within variance from random initialisation or sampling. No standard deviation across seeds is reported.\n- One big limitations is all the clusters are precomputed and it is very hard to generalize outside what you have already seend uring making the clusters.\n- How does DANCE handle actions that involve two coordinated agents (e.g., handshake) or tool–object interaction where the object itself moves (e.g., “pull-ups on bar”)? The current motion representation is strictly single human-centric.\n- Pose snippets are fixed-length (L frames). Actions whose discriminative cue lasts longer than L are split across several clusters and the bottleneck has no mechanism to model their ordering.\n- Skeleton co-ordinates are pixel-based, not camera-normalised. Without 2-D/3-D normalisation, clusters entangle action with camera geometry.\n- Motion clusters and GPT lists are dataset‐specific.Porting the model to a new domain (e.g., underwater sports apart from just sports like the examples) requires re-running both pipelines and retraining all heads which make it hard to follow the argument that “concepts give modularity”. The framework also does not work for anything different than one human doing some task in a exocentric view.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nDANCE, the method, inputs a video and aims to explainably classify the action occuring in said video. Each sample video during training also includes a binary presence/absence vector for concepts. \n\nA pre-trained encoder is used to extract features from the video. These features are projected with a linear concept layer into M activations (where M is the number of concepts), with each projection corresponding to a concept. This concept projection layer has weights that correspond to different types of concepts, specifically motion dynamics, objects, and scene concepts. The goal of this process is an explainable \"concept activation\" vector of size M. This vector is apportioned into a section related to motion dynamics, objects, and scenes. This vector is then used to predict a final action.\n\nPrior to training this method, concept discovery is done. To get motion related concepts, the authors use key clip selection to find chunks with motion cues. They estimate 2D pose and for each frame in a clip of length L and then cluster the motion patterns. After doing this across the training dataset, they can extract clusters as motion concepts. For the concept activation vector with activations related to motion, each index relates to these clustered motion concepts. For each training video, that video/clip has that motion concept annotated if it belongs to that cluster. \n\nBeyond motion concepts, object and scene concepts are generated for action classes by using VLMs. Specifically, an LLM first pseudo-annotates what objects are related to an action class, and then vision-language dual encoder is used to determine if those objects are present in the clip. Similarly to motion concepts, the clip is then annotated with the presence/absence of these objects through a similarity score between features. \n\nOverall, the paper finds ways to pseudo-annotate motion, object, and scene concepts in video clips. These pseudo-annotations come from discovered concepts in both the data and how the data relates to an LLM's understanding of what the actions should involve. After concept discovery, the presence of these concepts is represented as part of the sample.\n\nNext, training is done to predict the presence/absence of concepts independently of the action prediction. After this training is complete, the concept layer is frozen and the final classifier is trained. This separation of concerns leads to an interesting and interpretable way of understanding how concept activations lead to action classifications, as the concept scores are introspectable.\n\nSTRENGTHS AND WEAKNESSES:\nThis paper follows a tried and true methodology of data-centric concept discovery leveraged in the training of a classifier for use in a downstream task. The ability to pseudo-label annotate the concepts of the training data in order to first train a concept predictor and then action predictor is useful for creating an interpretable model. \n\nThe user study shows good performance for this method, and it is also useful quantitatively as it performs on par with the baseline w/o interpretability. \n\nRich interpretability as proposed in DANCE is useful both for better introspection into why/how classifiers make decisions and when a data-centric concept-learning phase as used in DANCE can actually help outperform uninterpretable baselines by grounding the focus of the classifier on certain relevant/salient distinctions (such as motion categories). Overall, this paper is quite interesting and helps expand the XAI domain. \n\nWeaknesses include the limitation of the motion-related concepts to human pose. Ideally, motion concepts could be more general and not just human-centric (although this is being used for action recognition).\n\nQUESTIONS:\nLLMs / VLMs were used for data annotation, couldn't they have been used for an additional baseline? It would be nice to see performance of more modern / more broad methods. \n\nAlthough language is a poor description of motion as shown in Figure 1, I'm curious how a method that was solely based on language would perform. Other prior work on relationship detection and other related action topics have shown strong priors based on the presence/absence of objects and/or just simply using language models on textual descriptions of the videos.\n\nLIMITATIONS:\nMotion clustering has implications for surveillance as it provides an easy data-centric way to extract concerning human poses. This type of easy-to-detect pose sequence clustering could have negative impacts as it is a lot easier to run this clustering on the outputs of a 2D pose detector over time than it is to process all the video in other more computationally expensive ways.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\n- This work proposes Disentangled Action and Context concept-based Explainable (DANCE) video action recognition\n    \n- The DANCE framework uses a frozen video backbone to extract features, then projects these features into a set of interpretable object, scene, and motion dynamics activations\n    \n- These interpretable concept activations are used as features for a linear action prediction layer\n    \n- To obtain a set of discrete motion dynamics concepts, they extract 2D pose sequences with a pose estimation model from the training videos and cluster the pose sequences. A video’s poses are then represented as a binary multi-label vector indicating which clusters the video’s pose sequences are members of\n    \n- To obtain object and scene concepts, they prompt an LLM to list the likely objects and scenes associated with each action class\n    \n- Training proceeds by first training the projection layers to predict the motion dynamics, then training a linear layer on the predicted concepts\n    \n- They find that DANCE’s explanations are heavily preferred to other interpretable baselines, and that classes are readily interpretable based on highly-weighted concepts\n    \n- The model’s interpretability allows for straightforward model editing, and they show that this process can improve classifications under distribution shift\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths\n\n- Straightforward, effective approach to an intrinsically interpretable action recognition model\n    \n- Results show a clear preference for their model over saliency-based methods and concept bottleneck models with spatio-temporal concepts, showing the benefit of pose-based interpretable features\n    \n- The writing is clear and the figures informative\n    \n- Qualitative examples are convincing of the approach’s explainability\n    \n- Manual model editing to avoid complete model retraining is a useful capability of this approach\n    \n\n# Weaknesses\n\n- The non-interpretable baseline in Table 1/line 296 is not described. How comparable is this model? Is it finetuned, zero-shot? [PapersWithCode](https://paperswithcode.com/sota/action-recognition-in-videos-on-ucf101) suggests that the top performance on UCF101 is 99.70, for example—significantly higher than the performance of the baseline w/o interpretability’s 88.4%. It is reasonable to focus on appropriate comparisons to DANCE, but more context for the baseline and SOTA models should be provided\n    \n- This method trades off flexibility for the sake of interpretability—adding a new class would likely require retraining the entire pipeline (e.g. mining concepts and clustering from scratch). A flexible but opaque alternative would simply be to use an open-vocabulary model. An explicit comparison with open-vocabulary, or more readily extensible alternatives, would strengthen the evaluations\n\nQUESTIONS:\nPlease see the weaknesses for changes I would like to see addressed (providing more context of DANCE's performance relative to baselines).\n\nFor clarification: It is unclear what “when this concept is deactivated” means on line 314. Is it assigned a weight of zero?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}