Title: Prompt-guided Disentangled Representation for Action Recognition
Venue: NeurIPS 2025
Accepted: yes
Keywords: Video Action Recognition，Disentanglement Representation Learning

Video Scene graph

Interpretable Graph Garsing
Abstract: Action recognition is a fundamental task in video understanding. Existing methods typically extract unified features to process all actions in one video, which makes it challenging to model the interactions between different objects in multi-action scenarios. To alleviate this issue, we explore disentangling any specified actions from complex scenes as an effective solution. In this paper, we propose Prompt-guided Disentangled Representation for Action Recognition (ProDA), a novel framework that disentangles any specified actions from a multi-action scene. ProDA leverages Spatio-temporal Scene Graphs (SSGs) and introduces Dynamic Prompt Module (DPM) to guide a Graph Parsing Neural Network (GPNN) in generating action-specific representations. Furthermore, we design a video-adapted GPNN that aggregates information using dynamic weights. Extensive experiments on two complex video action datasets, Charades and SportsHHI, demonstrate the effectiveness of our approach against state-of-the-art methods. Our code can be found in https://github.com/iamsnaping/ProDA.git.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a Prompt-guided Disentangled Representation framework for action recognition. Experimental results on video action recognition demonstrate the effectiveness of the proposed method, ProDA, compared to state-of-the-art approaches.\n\nSTRENGTHS AND WEAKNESSES:\nThe method achieves the SOTA performance on multi-label action understanding with interpretability.\nHowever, the novelty of the proposed method should be better refined and reorganized, and more comparison with existing methods and evaluation on other datasets should be classified, as shown in the questions below.\nMoreover, there are some typo mistakes such as \"State-of-the Art Methods\" in line 298.\n\nQUESTIONS:\n•The contributions and novelty of the proposed method should be better refined and reorganized to highlight its significance.\n\n• Existing multimodal large language models (MLLMs) have already demonstrated strong performance in action recognition, particularly in few-shot and zero-shot settings [1]. The authors should clearly state what advantages their method offers over such models.\n\n•In the proposed Action Disentanglement module, it is unclear whether the Action Specification captures temporal relationships. Would changing the order of specified actions or introducing interaction between actions affect model performance?\n\n•The idea of using prompts to augment video inputs and enhance model performance—i.e., prompt learning—has been widely explored in prior work. The authors fail to sufficiently discuss and cite relevant literature on this topic [3–5].\n\n•To validate the generalizability of the proposed method, additional widely used datasets such as AVA [2] should be considered.\n\n• The differences between VGNorm and previous GNN norm-based methods need to be clearly explained. If applicable, comparisons with more recent normalization techniques such as GRANOLA[6] should also be included.\n\nReference:\n1. LLaVAction: evaluating and training multi-modal large language models for action recognition\n2. Gu, Chunhui, et al. \"Ava: A video dataset of spatio-temporally localized atomic visual actions.\" Proceedings of the IEEE conference on computer vision and pattern recognition. 2018.\n3. Zhou, Kaiyang, et al. \"Learning to prompt for vision-language models.\" International Journal of Computer Vision 130.9 (2022): 2337-2348.\n4. Bahng, Hyojin, et al. \"Exploring visual prompts for adapting large-scale models.\" arXiv preprint arXiv:2203.17274 (2022).\n5. Khattak, Muhammad Uzair, et al. \"Maple: Multi-modal prompt learning.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2023.\n6. Eliasof, Moshe, et al. \"Granola: Adaptive normalization for graph neural networks.\" arXiv preprint arXiv:2404.13344 (2024).\n\nLIMITATIONS:\nThe author did not discuss the limitations of the work.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces ProDA, a framework that disentangles specified actions from multi-action videos using prompt signals and spatio-temporal scene graphs. The design is well structured, with components such as the Dynamic Prompt Module and a graph-based encoder that together achieve strong performance on standard benchmarks. The overall motivation is sound, and the system is practically useful for structured video understanding.\n\nHowever, the core idea of using prompts is only superficially explored. The prompts are implemented as simple multi-hot vectors without any semantic grounding, and the method does not incorporate language models that could reason about action semantics and generate richer prompt representations. This limits the conceptual depth of the work. While the prompt-guided framing is promising, its realization remains shallow. A more complete integration of language understanding could have made the contribution substantially stronger.\n\nOverall, this is a well-executed paper with a good idea at its core, but it lacks depth in exploring the full potential of prompt-based reasoning.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The paper proposes a disentangled representation learning framework (ProDA) for multi-label action recognition, with the goal of isolating specific actions from complex scenes. The motivation is clear and addresses a practical limitation of prior methods that process all actions jointly.\n\n2. The framework is well structured, integrating modules such as the Dynamic Prompt Module (DPM), Video Graph Parsing Neural Network (VGPNN), and a novel normalization strategy (VGNorm) tailored to spatio-temporal scene graphs.\n\n3. The empirical results are strong, with consistent improvements over existing methods across multiple benchmarks. The ablation studies are thorough and support the utility of each component.\n\nWeaknesses:\n1. Although the paper frames the method as “prompt-guided,” the prompts are implemented using simple multi-hot vectors that lack semantic richness. This shallow encoding limits the model’s ability to reason about unseen or open-set actions, as it cannot generalize beyond the fixed label space.\n\n2. The method does not incorporate any language model or text encoder to provide semantic grounding for action prompts. As a result, the full potential of prompt-based guidance is underutilized, and the framework misses the opportunity to leverage external knowledge or compositional semantics.\n\nQUESTIONS:\n1. The key concern is encoding prompts with fixed multi-hot vectors. This choice appears to block open-set recognition and prevents the framework from exploiting prior knowledge available in large language models, thereby limiting the broader significance of the contribution. Nevertheless, the work presents a well-executed and novel perspective on action recognition. If additional analysis is provided, such as evaluating the potential of language-model-guided prompts or demonstrating the flexibility of the current encoding scheme, I would be happy to raise my rating.\n\n2. Another concern relates to the fusion strategy used in Equation 1, where visual features and prompt features are combined through simple addition. This choice may limit the expressiveness of the interaction between action specifications and scene graph features. It would be valuable to explore alternative fusion methods such as concatenation followed by a linear layer, or cross-attention mechanisms. A comparison of these approaches could provide insight into whether the current additive design is optimal or if stronger interaction modeling would further enhance performance.\n\n3. VGNorm computes temporal statistics using mean-based aggregation to enforce consistency across frames. While this design is standard in normalization, it remains unclear whether mean alone is the most effective choice for modeling temporal dynamics in spatio-temporal scene graphs. Mean aggregation may dilute salient or bursty features that occur sparsely over time. It is not clear whether combining mean with other statistics such as max, or using a learnable aggregation function, would offer any advantage in this context. Could the authors clarify whether such alternatives were considered, and whether there is theoretical or empirical justification for relying solely on mean-based aggregation?\n\n4. The proposed method relies on object bounding boxes to construct spatio-temporal scene graphs, which serve as the foundation for downstream disentanglement. However, the quality of these bounding boxes during training—including their number, category diversity, and spatial distribution—is likely to have a significant impact on the effectiveness of feature disentanglement. The paper does not provide any analysis on how variations in bounding box quality influence model performance. Has this factor been studied, and is there any quantitative or qualitative assessment of how sensitive the method is to the quality of the input object annotations?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 4, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a framework for disentangling specified actions from multi-action video scenes using prompt-guided spatio-temporal scene graphs. By leveraging prompts to focus on particular actions, the method isolates relevant object and interaction representations, making it easier to model complex scene dynamics. This targeted disentanglement leads to improved performance on standard multi-label action recognition benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths : \n\n1. The paper is well written, and the motivation, key ideas, and contributions are clearly articulated and easy to follow.\n\n2. The idea of disentangling desired actions by leveraging prompts is very compelling. Based on the results provided, the approach appears to be effective. While vanilla methods may struggle in ambiguous scenarios, the proposed DPM framework seems to overcome these limitations by generating adaptive prompts conditioned on both the action specification and the video content.\n\n3. The ablation study is thorough and clearly isolates the contributions of each module, helping to attribute the performance gains to specific components of the framework.\n\nWeaknesses :\n\n1.  While the method shows some improvements in mAP, the added complexity of the proposed approach may not be fully warranted for the action recognition task, particularly in the absence of a detailed comparison of model efficiency or computational cost. In other words, can some of the performance gains be coming from increased model capacity rather than the disentaglement loss? \n\n2. Since the central claim of the paper is that explicit disentanglement of actions improves performance, a more compelling demonstration would involve applying the disentanglement loss to an existing baseline model without increasing the parameter count. This would better isolate the effect of the disentanglement itself. As it stands, the comparison in Table 3 involves different architectures and training setups, making it unclear whether the performance gains are due to the disentanglement mechanism or other architectural differences and whether the disentanglement loss is always strictly necessary for all models. \n\n3. The evaluation is limited to a single dataset, and the performance gains over certain baselines are relatively modest. Including results on one or two additional datasets would strengthen the generalizability claims and further validate the robustness of the approach.\n\n4. Although the core ideas are well-motivated, the transition between some concepts could be smoother. Improving the overall narrative flow would help the reader better follow the progression of the proposed framework.\n\nQUESTIONS:\nAlready mentioned in weaknesses\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the challenge of accurately recognizing specific actions in complex, multi-action video scenes, where conventional methods typically extract unified features and struggle to capture action-specific semantics and inter-object interactions. To tackle this, the authors propose a novel framework called ProDA (Prompt-guided Disentangled Representation for Action Recognition), which enables the model to disentangle and analyze any specified action within a scene by leveraging Spatio-temporal Scene Graphs (SSGs) and prompt-based conditioning. Specifically, ProDA introduces a Dynamic Prompt Module (DPM) to generate action-aware prompts, a customized Video Graph Parsing Neural Network (VGPNN) to extract action-relevant subgraphs, and an Action Disentanglement Loss that guides the network to learn semantically meaningful and reconstructable representations. Without requiring explicit action localization labels, ProDA achieves state-of-the-art performance on multi-label action recognition benchmarks while maintaining strong interpretability and localization capabilities.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The paper is motivated by a clear and relevant problem — the difficulty of recognizing specific actions in multi-action scenes using unified features. \n2. The writing is well-structured and easy to follow. \n3. The proposed method does not require action localization supervision.\n\nWeaknesses:\n1. The paper lacks analysis of the model's inference efficiency, which is important given the complexity of the proposed pipeline. \n2. While the motivation for action-centric feature disentanglement is sound, the paper does not provide visual evidence (e.g., t-SNE) to demonstrate the effectiveness of the disentangled representations compared to conventional unified features. \n3. Evaluation is limited to a single dataset, raising concerns about generalizability. \n4. Additionally, the potential impact of SSG quality on model performance is not analyzed or discussed, which could affect robustness in real-world applications.\n\nQUESTIONS:\n1. Can the authors provide inference speed or computational cost analysis?\n2. Could the authors include a qualitative visualization (e.g., t-SNE) comparing disentangled vs. unified features?\n3. Why is the evaluation limited to a single dataset? \n4. How sensitive is ProDA to errors in SSG construction?\n\nIf these issues are properly addressed, I would be happy to raise my evaluation score.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-09"}