Prompt-guided Disentangled Representation for Action Recognition
Authors: tianci wu
Guangming Zhu
Lu jiang
Siyuan Wang
Ning Wang
Nuoye Xiong
zhang liang
Venue: NeurIPS 2025 poster ()
Keywords: Video Action Recognition，Disentanglement Representation Learning
Video Scene graph
Interpretable Graph Garsing
Primary area: deep_learning

Abstract: Action recognition is a fundamental task in video understanding. Existing methods typically extract unified features to process all actions in one video, which makes it challenging to model the interactions between different objects in multi-action scenarios. To alleviate this issue, we explore disentangling any specified actions from complex scenes as an effective solution. In this paper, we propose Prompt-guided Disentangled Representation for Action Recognition (ProDA), a novel framework that disentangles any specified actions from a multi-action scene. ProDA leverages Spatio-temporal Scene Graphs (SSGs) and introduces Dynamic Prompt Module (DPM) to guide a Graph Parsing Neural Network (GPNN) in generating action-specific representations. Furthermore, we design a video-adapted GPNN that aggregates information using dynamic weights. Extensive experiments on two complex video action datasets, Charades and SportsHHI, demonstrate the effectiveness of our approach against state-of-the-art methods. Our code can be found in https://github.com/iamsnaping/ProDA.git.

Prompt-guided Disentangled Representation for Action Recognition
Tianci Wu, Guangming Zhu, Jiang Lu, Siyuan Wang, Ning Wang,
Nuoye Xiong, Zhang Liang
School of Computer Science and Technology, Xidian University
(tianciwu, lujiang, siyuanwang, ningwang, nyx/@stu.xidian www.iiiddiiiii.... edu. {gmzhu, LiangzhanggOxiddan.eeeee
Abstract
Action recognition a fundamental task in video understanding. Existing methods typically extract unified features to process all actions in one video, which makes it challenging to model the interactions between different objects multi-action scenarios. To alleviate this issue, we explore disentangling any specified actions from complex scenes as am effective solution. In this paper, we propose Prompt- guided Disentangled Representation for Action Recognition (ProDA), novel framework that disentangles any specified actions from a multi-action scene. ProDA leverages Spatio-temporal Scene Graphs (SSGs) and introduces Dynamic Prompt Module (DPM) to guide a Graph Parsing Neural Network (GPNN) in generating action-specific representations. Furthermore, we design rideo-adapted GPNN that aggregates information using dynamic weights. Extensive experiments on complex video action datasets, Charades and SportsHHI, demonstrate the effectiveness of our approach against state-of-the-art methods. Our code can be found in https://githutt cww. httpa//c.../.aaaaaaaaa////...o..
1 Introduction
Action recognition is one of the fundamental tasks in the field of video understanding [16]. and it remains an active topic in the vision research community. Current mainstream methods can be categorized into two types based on their data processing approach. The first type is based on complete video data. These methods typically input the untrimmed video into 2D Convolution Neural Network (CNN), 3D CNN, or Vision Transformer (ViT) 7 using the network to extract features and then performing classification using the extracted features [10]
27 30 (27) The second type is object-centric, where object detectors first crop objects from the video and then input them into ViT Graph Neural Network (GNN) 553
45
57
48
55
48
5
Some methods (59 20
47 35
also construct Spatio-temporal Scene Graph (SSG) from these objects and then feed them into the network. However, these methods are essentially information extraction processes that handle all actions in the video at once, making it difficult to model the interactions between objects in complex scenes 48 (55]. In this paper, we aim analyze actions with a disentangled approach. For instance, in video with multiple actions, we want the network to focus on analyzing some specified action(s). Moreover, we construct SSGs for this video and aim to disentangle the subgraph, which consist of the specified action-relevant nodes, enabling better understanding of the actions (as shown in Figure D.. Existing Disentangled Representaion Learning (DRL) methods typically approach disentanglement from different perspectives. Some methods focus on attribute disentanglement, aiming to separate content from background [2]. Others adopt an information perspective by disentangling representations into structural invariants and local variations (e.g., shared information
"Corresponding Author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Promutt IW TV. Taking Phone]
Multi-Hot Vector
Prompt Generator
Frameniateed Predit tome Watch Takong Phone
Action Specification
TV
Specitied Actions
Human
Phone
Human
Disetlangloment Network
Human Door
Building, SSG
relne-rebted Predkctions
TV
Human
Door
Wallen (hrough Dourway Cleing Door Other Actions
Dourway
Phone Door Way
Action Disentanglemeut
Our A ction Disentanglement method
Vidco
Building SSG
Human Door
Humann Deyr
Video Enoader
Huma Door
El Dooe
Watch TV, Taking Phone Willing Through Doorway Closing Door
Prodictons:
Phont Dourway
Pholle
Fealure Extractio
Video
(b) Existing feature extraction methods
Figure I: Comparison between (a) our proposed Prompt-guided Representation Disentanglement method and (b) existing methods using SSG. Existing methods treat action modeling feature extraction process, handling all actions in one video. In contrast, our method disentangles the action representation into two parts: one for the specified action and the other for the remaining actions, enabling more focused analysis and better action understanding.
34 From a structural perspective, some approaches disentangle graphs into a fixed number components(e.g, latent representation, subgraphs) [26 25 24 6 (88 [2]. A task perspective has also been explored, where complex tasks are divided into multiple subtasks to facilitate more structured representation learning [29]. action recognition, we argue that disentanglement should be approached from the action perspective, enabling the dynamic separation of semantic factors related to any specified actions. Specifically, we expect the network to analyze any actions specify and to disentangle the corresponding subgraph from the whole SSG, cnabling more targeted and interpretable action understanding.
across nodes and private, node-specific features) 6 58
To this end, in order to enable the model to disentangle representations conditioned on any specified action, we draw inspiration from recent advances in prompt tuning [3 Hromptggudded Disentangled Representation for Action Recognition (ProDA). As depict in Figure the
39
21 56
and propose
(a), ProDA disentangles the SSG into two complementary parts: one corresponding specified actions, and the other to the remaining actions. To achieve this, we construct an action- aware disentanglement network that can disentangle any specified actions from SSG based on prompt guidance. Specifically, introduce the Dynamic Prompt Module (DPM), which generates action- aware prompts by combining the multi-hot vector of specified actions with the SSG features. These prompts guide the Graph Parsing Neural Network (GPNN) to perform action-awan disentanglement. Furthermore, we propose a Video Graph Parsing Neural Network (VGPNN), a GPNN variant tailored for SSG, to perform action disentanglemenn. To ensure temporal consistency and preserve the differences across SSGs from different videos, we introduce a Video Graph Normalization (VGNorm) module, which is applied after each VGPNN layer. Finally, introduce Action Disentanglemest Loss (AD Loss) to supervise the separated representations. To ensure that the disentangled features retain essential information, disentanglement rather than mere feature extraction.
incorporate reconstruction term, promoting true representation
In summary, the major contributions of this work are follows: 1) We present novel frame- work ProDA, which disentangles any specified action from multi-action videos, enhancing the interpretability of action understanding: 2) Our method achieves State-of-The-Arr performance multi-label action recognition benchmark. 3) Without requiring action localization supervision, our method demonstrates strong action localization capabilities, further validating its effectiveness and interpretability in understanding complex actions.
2
Related works
Existing action recognition methods can be broadly categorized into three types. First type is CNN- based methods. To compensate for the lack temporal modeling CNNs, two-stream networks introduce an additional temporal stream (e.g., optical [10] 9
49 By jointly modeling

---

Racondmucrd
Sputial Coniial-
Readout
CLiP Image
Reponstmuttion Network
Encoder
RolAlign
DPM
UAP
&CLIP
Readout
Flaee
RotAlign CCIP
Encoder
Shared
Cas
Video
Spettal-errpera Scone Gruphs
Resdout
Readout
4,
Object Bounding Box
Sase
Figure 2: An overview of our ProDA. We first convert the input video into a sequence of scene graphs, which are then processed by Video Encoder. Next, generate two complementary prompts from the Action Set (SAP and UAP) using DPM to guide the VGPNN in disentangling the actions into two complementary SSGs, each containing distinct parts of the action. These disentangled SSGs, along with their fused version, are fed into corresponding readout blocks to obtain the global representations of partial and complete actions, respectively.
spatial and temporal features, 3D CNNs are able to better capture spatio-temporal information and thus yield improved performance 42 (I). The second type is ViT-based methods. Most methods
either treat entire frames as input tokens or divide each frame into patches before feeding them
into the model [13 45 27 30 27]. These methods treat the video as a whole, which makes
difficult to model interactions among objects. and often introduces substantial noise. Our work alleviate this by disentangling actions from the SSG, enabling more precise modeling of object interactions. The third type is Objected-centré methods. Object-centric methods construct more compact and efficient video representations by focusing on regions of interest (e.g., objects in the videa) [53], and enabling generalization of unseen interaction relationships through structured 57 48 59]. Nevertheless, these methods remain limited in their ability
representations [53 33
to effectively model interactions among objects. The incorporation of SSG [59] 20 effectively addresses the difficulty in modeling object interactions. In scene graphs, videos are represented as spatio-temporal graph structures, with objects serving as nodes (e.g., human, cup) and their interactions or spatial-rclationship (c.g., holding, above) modeled as edges. (35] processes the scene graph of each frame using Graph Neural Network (GNN). On the other hand, [47] flattens the scene graphs and feeds them into a ViT 771, leveraging the long-range modeling capability of self- attention (3 effectively capture interaction transitions. These SSG-based methods still adopt feature extraction approaches that process all actions in a video, which makes it challenging to model interactions between different objects in multi-action scenarios. In contrast, method disentangles actions from complex scenes, enabling better understanding of the actions
Methodology
Our method aims disentangle any specified actions from multi-action scenarios. This disentan- glement provides more concise feature representation for analyzing the specified actions, leading to improved action understanding. The overall architecture of our method is illustrated in Figure [ first convert the input video into SSG. This graph is then processed by a video encoder (specifically, Dual-AI (15)) model temporal and spatial interactions among nodes within the SSG. Next, guided by the DPM, the video features are fed into VGPNN for representation disentanglement Finally, to further enhance the disentanglement, in addition to standard classification supervision, we introduce both reconstruction loss and a disentanglement loss as constraints. We generate the SSG (38) using existing methods and adopt off-the-shelf video encoder [5]: the following focuses on the remaining components of the network architecture illustrated in Figure
3.1 Formulation
In SSG, each frame of a video processed into a graph, where nodes represent the objects present in that frame, and edges represent the semantic relationships between these objects (c.g., spatial or
3

---

interaction relations) Formally, let G (V,E, T,E,R, R) denote the complete SSG constructed from the video, where V is the set of all object nodes across all frames, E is the set of all directed edges, and R. is the set of all relation types. Each node v E V is uniquely indexed from {1 ......; [/]. Each edge represents a directed semantic relation of type r E R. from node v to node w. Let 9: (V,, & R.) denote the scene graph of the i-th frame, where V; S V, & C & and R2 C R denote the frame -eeeni--eeffieeeee object nodes, their edges and relations. We denote oy as the j-th object node in frame i, and ea* as the edge from vi to up with relation type r.- E R4. The node features can be denoted as Frode = U1 12 -I.....,NN. E RRNxXxD where / E R represents the feature of node vs in the j-th frame. Here, N, M, and D denote the number of frames in the video, the number of nodes per frame, and the dimensionality of each node feature, respectively.
e e= (U, w. r) E
3.2 Dynamic Prompt Module (DPM)
The DPM is key component of our method. Since action disentanglement is driven by prompts, ambiguous prompts can make it hard for the model to focus on the specified action-relevant regions, leading to failure in disentangling the specified actions. we generate dynamic prompts based on the node features of the SSG, which guide the model to attend to action-relevant nodes and achieve better disentanglement. In this section, we introduce how prompts are generated based on arbitrary action specifications. We first describe the design of action specifications, followed by their construction strategy. Finally, we show how prompts are generated from the specifications, which guide the VGPNN to disentangle actions. Inspired by [8 31]. we inject learnable prompts into SSG feature space to accomplish guidance.
Action Specification Design. We first represent each action category as a one-hot vector. To specify combination of actions to disentangle, construct multi-hot vector (e.g.. [1, 0,............ ... 1)), where each indicates a hot element, i.e., a specified action. By aggregating several ooe-hot vectors, forming the Action Specification (AS). In designing the AS, we consider two components: one for the actions we intend to disentangle, called the Specified Action Prompt (SAP), and another for the actions we aim to exclude, called the Unspecified Action Prompt (UAP). For the design of SAP, a straightforward approach is to directly input the label- -bsedd SAP into the model. However, this can lead to shortcut learning, where the model relies on label cues rather than genuinely parsing the video content. Additionally, in many real-world scenarios, the exact set of actions in a video is not always known. To address these limitations, we further design an alternative version of SAP that includes both the ground-truth actions and distractor actions (i.e., actions not present in the video). This encourages the model to focus on learning discriminative features rather than simply following label hints. For the distractor-injecter SAP design, we keep the number of hot elements in the SAP multi-hot vector to K, only some of which correspond to ground-truth actions, while the rest are distractors. To prevent the model from exploiting shortcuts, we construct the UAP by inverting the SAP. In this way, the combination of SAP and UAP covers the entire label set, ensuring a balanced and comprchensive specification of both present and absent actions (For better clarity, we include illustrative examples in Sec. C.I. In certain cases, SAP and UAP may contain only distractors (i.c., no true actions). For such scenarios, we introduce an auxiliary label called "no-action" to explicitly supervise the model.
Action Specification Construction. The AS consists of two types: cetrrccoorinjecter and non- cetrrcttor-ijected. The distractor-ijjected AS includes combination of present labels (i.e., labels of some actions present in the video) and absent labels (i.e., labels of some actions absent in the video). Enumerating all possible label combinations is computationally infeasible. To address this, we adopt random sampling strategy. For video with L ground-truth labels, the distractor-injectéd SAPs, we generate L + different multi-hot vectors with a length K. Among them, i-th SAP contains i random sampled present labels, and K - i random sampled absent labels. In contrast, the non-distractor-injeeeed SAPs, we generate L different multi-hot vectors, only contains random sampled present labels. UAPs are then obtained by inverting each SAP. This results in 2L + 1 different pairs of SAP and UAP. This design enables the model to learn from a diverse set of action combinations, promoting robust disentanglement by exposing it to both clean and noisy prompts while maintaining efficiency. For better clarity, we include illustrative examples in Sec. C.2
Prompt Generation. A possible way to incorporate AS (symbolized as v) is to project it into the same dimension space as the node features using a linear layer and add directly to the features, we refer to this method as SimplePrompt. However, due to the static essence of such prompts, this

---

approach may struggle to capture contextual variations and risks encouraging shortcut learning, particularly in our dynamic disentanglement setting, where guidance need adapt to varying action contexts. To address this, we propose the Dynamic Prompt Module, which generates an adaptive prompt pi to each node feature f. The prompt Fº is constructed as linear combination of candidate prompts {qI, (91,92,... .2.....977)). (r), which are generated from v. The combination weights Ap = {a1 t.j &. ,...,00T] 1,7) } are computed based on both v and f.- It can be expressed as:
AP = o("" -W"
v f()))
(1)
The resulting prompts is then added to Frode to generate prompted feature Rader which replaces Fnode in subsequent processing. It can be expressed as:
T
Frode = Pl. + pi. pi.....,
SM + )
1 a2
qt
t=
3.3 Video Graph Parsing Neural Network (VGPNN)
We aim to disentangle subgraphs corresponding to specified actions. To achieve this, we require a network that, under the guidance of prompts, can parse action-relevant subgraphs by directionally propagating action-specific information. Inspired by GPNN (37], we design a variant for SSG, called VGPNN, GPNN parses action-relevant nodes by generating dynamic weights that control the direction of information flow. It consists of three key steps: Link, Message, and Update. the Link step, it constructs edges between the same object nodes across different frames and learns aggregation weights for each timestamp. In the Message step, it multiplies these weights with messages, where each message is composed of node's 's features and its edge features. In the Update step, it selects nodes from a specific frame as the initial hidden states, and updates them sequentially using the weighted messages from different time steps. However, GPNN has several limitations when applied to SSGs. I)It only updates a node from a single frame, ignoring joint modeling of nodes across the entire video. 2) It performs shallow temporal modeling, as the learned weights are used only once for aggregation and are not further refined or propagated through deeper temporal structures.
Video Graph Parsing Neural Network (VGPNN). VGPNN is designed to disentangle features by decomposing a SSG into two subgraphs, guided by a given AS. To achieve selective information aggregation, with the same to GPNN [37]. each VGPNN layer processes the entire SSG in three sequential steps. In Link step, we perform temporal modeling of edges e {cf e.Ah. ...... eye via self-attention [3 over the same interaction across time. In Message step. we generate dynamic weights from the edge features, which are then multiplied with the messages, where each message is composed of node's features and its edge features. In Update step, we update all nodes by adding the weighted messages to the corresponding nodes directly. A MLP is then used to fuse the updated and original node features. More details please refer to Sec. B
Video Graph Normalization (VGNorm). SSG exhibits characteristics of both graph structure and spatio-temporal properties of videos. However, many commonly used normalization methods tend to focus on only one aspect. For instance, BatchNorm [18] primarily normalizes across the temporal batch dimension, while LayerNorm I operates along the node (spatial) dimension to enforce spatial consistency. This one-sided focus limits network performance. To ensure temporal consistency and preserve the differences across SSGs from different videos, our method focuses on two key aspects. First, we use the frame-wise global mean Mg R.V to ensure temporal consistency across frames. During training, we compute the mean μ R.N for each batch in real-time and update the llg E R using a coomentum-based approach. Specifically, for batchsize B nodes features Fnode- = {F FFooee, ..... ,........ FFodd) } E BxBxxxxxxxx where Frode E ERN NxMx is the b-th video nodes features, the global mean is computed as:
M ΣΣΣ
Mg = 0m Mg + (1 (m)).
# == B.
node
(3)
b=0 J=0
where (m represent the momentum. This global mean is used to normalize the features at each frame, ensuring temporal consistency.
Second, we introduce mean scale shift a e RN to account for the heterogeneity across different video SSGs. For the node features Fridee of the b-th video, the normalization is defined as:
mnode
2 Emmo >b (FFode 22g
F = 7
a Pg 3
(4)
M D
5

---

Here, 7 E R' are learnable affine parameters, and F3 is the normalized feature that replaces Fo. in subsequent processing.
Readout. First, we compute dynamic weights for intra-frame nodes to aggregate them into frame- level features. Then, calculate frame-level weights for each frame feature to perform temporal aggregation, yielding a global representation of the SSG, which is used for subsequent action classification supervision.
3.4 Optimization
In this section. we present the loss functions used to optimize our framework. Our design includes two complementary objectives: an Action Classification Loss, which guides the model to correctly disentangle features as specified by the AS, and an Action Disentanglement Loss, which ensures that the resulting representations are well separated. Together, these losses encourage both accurate action recognition and effective disentanglement in multi-action video scenes.
Action Disentanglement Loss (AD Loss). AD Loss includes a reconstruction loss and a disentan- glement loss. For disentanglement, we treat the two disentangled representations as random variables and minimize their Pearson's 's correlation coefficient enforce disentanglement. However, complete separation may discard shared semantics (e.g., common objects or interactions). To preserve these, we introduce margin-based relaxation using a ReLU, allowing reasonable overlap while penalizing only excessive correlation. For a mini-batch with batchsize B, disentanglement loss Cais can be expressed as:
1 Σ ReLU (Co( 1h (FF))
Cdss
m)
(5)
B
(Var(os(7))
(er
where F3, R represent the action representations disentangled by UAP and SAP of b- th video, respectively. Φb, th,, are measurable functions [1. Cor(-), Var(-), my represent covariance, variance and margin, respectively.
To ensure that the model performs disentanglement rather than feature extraction. We reconstruct the original feature F E R.N xM which is obtained from the video encoder before disentanglement. And we fuse the features FG and then obtain the reconstructed representations. Fi E NR with the reconstruction network. The reconstruction process can be expressed
F Ne F + (1 - 5) F.)
8= Netaa(FF,
(6)
where Net1, Nety represent reconstruction and fusion weight generation functions. And apply Mean Squared Error to enforce this constraint. Since explicit guidance is added. we were to strictly align F3 with F%, the video encoder might focus aligning with the prompts instead of the video. This could result in the video encoder extracting irrelevant information, causing the model to output results based on the prompts instead of the video. To prevent this, reserve space by applying margin- based ReI.U relaxation m2 to tolerate overlaps, reconstruction loss Cree can be expressed as:
Cree
Σ ROLU(I - - b-0
B
Overall. In ProDA, we predict the labels of the disentangled actions specified by UAP and SAP, denoted as au a, as well as the fused action label (i.e., the video label). The first two predictions ensure that the model correctly disentangles representations according to the AS, while the fused prediction encourages information completeness and promotes interaction between the disentangled components, During training, we have access to the ground-truth labels for UAP and SAP, denoted as Va. Bw. along with the video ground-truth label yt- Therefore, we can compute the total loss L as:
Lace (uu, yu) + A2 Loce (aw, yw) + As Loce (as: 31) + Aa (Lais + Cree)
(8)
actions classification loss
action disentangl gennent
where, Lice binary cross-entropy loss.
Experiments
4.1 Experiment Setup
Dataset. (1) The Charades dataset 40 consists of 9,848 videos, each with an average duration of 30 seconds. It covers 157 action categories, with each video containing an average of 6.8

---

Table 1: Multi-label action recognition performance comparison on the Charades's validation set in term of mAP. SSG:ground truth SSG. Bbox: bounding box from ground truth SSG.
model
pretrain
Modality
mAP
model
pretrain
Modality
mAP
SGFB+NL 20
K400
RGB
44.3
SGFB+NL 20
K400
RGB4SSG
60.3
13D4NL
4
K400
RGB
37.5
LalAR
47
K400
RGB+Bbox
63.6
OR2G+NL
K400
RGB
44.9
OR2G
K400
RGB+SSG
63.3
LL
47
K400
RGB
45.1
LaLAR+NL [47
K400
RGB+Bbox
67.4
(iCTR(B/16)
23
CLIP
RGB
50.1
OR2G+NI. [35
K400
RGB+SSG
67.5
VicTRR(L/14)
23
CLIP
RGB
57.6
Ours(B/16)
CLIP
RGB+SSG 71.1
Table 2: Human-Huuan Interaction Classification results in SportsHHI. SportsHHI is our reimple- mentation based on the official SportsHHI code, with minor modifications (e.g., parameter size) to match the capacity of ProDA for fair comparison.
method
mAP
R 150
R@ 100
R@:00
R
Params
STTran
3.31
71.29
62.91
42.67
22.14
HORT
19
3.75
78.57
67.78
50.33
26,96
SlowFast
5,00
81.66
74.00
52.74
26.82
ACARN
36
5.44
82.85
75.22
56.53
31.77
SportHHl [50] (baseline)
10.69
89.25
82,93
68. 13
43.72
111.18M
SportsHHI" (reimplctmentee)
10.16 (10.53)
90.07 (70.82)
84.83 (11.9)
68.40 (10.37)
40.16 3.56)
123.1
Ours (baseline+ProDA)
13.55 (12.66)
92.54 (?3,29)
87.91 (14.98)
72.69 (14.56)
47.81 (14.09)
123.19M
distinct actions. Multiple actions occur simultaneously, making the recognition task particularly challenging. Based on Charades, the Action Genome dataset (20) provides tree-graied annotations by decomposing actions and focusing video clips where the actions take place. It contains 234K
keyframes, with annotations for 476K object bounding boxes and 1.72 million object relationships.
(2) The SportsHHI dataset (50) is specially designed dataset focusing on human-human interaction in sports. It covers basketball and volleyball games, and video data are selected from MultiSports (28). This dataset labels 34 interaction classes with 118,075 human bounding boxes and 50,649 interaction instances. Following (0)), we conduct evaluations on Charades for multi-label action recognition and on SH1 for Human-Human interaction Classification (HHICIs), adopting mean Average Precision (mAP) as the metric.
Experiment Setup. (1) On Charades, all experiments are conducted under the same design. For each video, the network takes N frames as input. During training, frames are randomly sampled from the video, while during validation, the frames are extracted evenly over the whole video. In the first stage, the entire network trained end-to-end, and the resulting weights are used to initialize the second stage. In the second stage, only the classifiers for au, as. 0g trained. Additionally, to enable deeper analysis, we introduce an auxiliary prediction am, which is inferred from the features before disentanglement (features from Video Encoder illustrated in Figure (2). Its corresponding classifier is also trained during the second stage. (2) On SportsHHI, we adopt the official open-source model and training code without altering any training settings. For fair evaluation, we conduct two experiments: 1) extending the official baseline with our proposed modules, namely the DPM and AD Loss; and 2) constructing a parameter- matched baseline by enlarging its model size match ours, without incorporating any proposed components.
4.2 Compared with Statc-of-The Art Methods
We compare the action recognition accuracy of the proposed method and the State- of-The-Art methods (SoTA) on the Charades. Table presents our results on the Charades dataset. ProDA achieves state-of-the-an performance. Our method outperforms several CNN- and ViT-based approaches, including 13D [ and VicTR (23). This demonstrates that our approach can better capture object interactions within structured data (e.g.. SSG), enabling a deeper understanding of video. Notably, our method with CLIP-B/16 (38 still surpasses VicTR (L/14) 23), even though we only use an image encoder, while VicTR 23 additionally employs a text encoder. Furthermore,
Charades.
our method also outperforms several approaches that leverage SSG 35 47. This demonstrates that explicitly disentangling and processing actions separately leads to more accurate understanding of complex actions. It is also worth noting that our method achieves this performance without requiring any additional external models.
7

---

Table 3: Ablation of prompt
Table 5: Ablation study of loss function on Charades dataset.
prompt type
a,mAP (%)
Gm. (%)
G denotes the baseline training only the video encoder with- out ProDA; denotes using ProDA with only the action
SimplePrompt
68.07
57.80
DPM
71.10
59.04
classification loss.
Table 4: Ablation of normalization
Eres w/o margin
Crea
Luis w/o margin
Charades (5))
mAP on
fun mAP on Charades (%)
norm type
a,mAP (%)
ag MpP (%)
67.45
57.38
68.43
8.03
LayerNorm
68.12
$7.96
67.78
58.28
BatchNorm
65,39
58.20
2
64.89
57,90
GraphNorm B
68.61
58.32
71.10
59.04
VGNomm 71.10
59.04
58.01
SportsHHI. We compare our method with the SoTA baseline and its parameter-inereased variant, as reported in Table 2 Enlarging the baseline leads to 10.8% increase in parameters, whereas our proposed DPM introduces only 1.46M additional parameters (1.18% of the total). Despite such a small overhead, our method achieves notable improvements across all metries. Even under comparable parameter settings, it consistently outperforms the baseline. These results demonstrate the strong generalization capability of our approach and confirm that the performance gain primarily arises from the proposed AD Loss rather than from increased model capacity.
4.3 Ablation Study
Impact of DPM. DPM is key component of our framework. To verify its effectiveness, we conduct comparisons with both Simple Prompt and DPM. In addition to evaluating the final fused output as we also train separate classifier on am, the features before disentanglement, for more comprehensive analysis. The results in Table B show that using DPM improves a, by 3.03% mAP and am by 1. 1.24% mAP compared the SimplePrompt. These findings indicate that our dynamically generated prompts, conditioned on the input, better capture the evolving nature of actions in videos, enabling more effective representation disentanglement. Furthermore, the clearer disentangled features also enhance the performance of the video encoder network.
Impact of VGNorm. We design VGNorm to adapt to SSG, which integrate both video and graph structures. We compare it with several widely used normalization methods, and observe that VGNorm achieves the best performance as shown in Table 4 Compared to BatchNorm [8]. VGNorm better preserves the spatial structure of graphs while maintaining temporal consistency, leading to a significant improvement of +5.71% % mAP. Compared to LayerNorm ①1. which normalizes features independently and ignores the graph structure, VGNorm incorporates spatially consistent mean scaling to preserve temporal consistency and capture the heterogeneity across SSGs from different videos, resulting in a +2.98% mAP improvement. Finally, compared to GraphNorm temporal consistency, enabling the model to better capture temporal dynamics, which contributes to +2.49% mAP improvement. Moreover, by jointly modeling both temporal consistency and the heterogeneity across SSGs, VGNorm produces more stable and adaptive representations. This leads to the best performance on am,, further demonstrating the effectiveness of our normalization strategy.
3. VGNorm enhances
Impact of AD LOss. We design ADLoss enable effective feature disentanglement by introducing a margin-based reconstruction loss and a disentanglement loss. As shown in Table SS removing the margin constraint results in higher d, performance, but the accuracy of am drops sharply to 8.03% mAP, indicating that the video encoder fails to retain meaningful representations. This suggests that, without the margin, the model learns take a shortcut by relying solely on the prompt to generate features, bypassing true video understanding. In contrast, when only dis- entanglement is applied v.s. 3) the model still achieves notable improvements, demonstrating alone can help capture object states and interaction transitions more
(0 v.s. @)
that action disentanglemen effectively. Comparison between v.s. ! further shows that treating all actions as whole leads to suboptimal results, as the model cannot preserve the independence of different actions. However, the disentanglement structure under reconstruction constraints pushes the video encoder extract richer features, thereby improving its performance. Moreover, o V.S. ① shows that adding reconstruction transforms the process from mere feature extraction to genuine disentanglement, preserving more interaction cues. Finally, I v.s. 2 confirms that combining reconstruction with disentanglement ensures the independence of action representations, allowing the model to more accurately capture the dynamic interactions within video, ultimately achieving superior performance.
8

---

Action Watching someltinggsommonn//hcmmssssss mirror
Action Drinking from cupíglass/bottle
Figure Frame-level weights s predicted by the readout function for different non- con-distractor-injee SAPs within the same video. The heatmaps (colored bars) visualize the predicted weights, reflecting the relevance of each frame the target action. The red bars indicate the ground-trutt action segments, while the green bars denote our localization results obtained by thresholding at 0.7.
Table 6: Non-distractor-injeeeed SAP. label Table Distractor Distractor-injected SAP. label nums indicates the length of the ground-truth indicates the length of the ground-truth label set for each video
label set for each video
lahel nums
o, mAP
label nums
2, mAP
label nums
"g mAP
label nums
ag mAP
100.00 +0.0
9
99.80 #0.3
100.0 =0.0
9
85.63 +6.6
100.00 10.0
99.45 =0.6
2
90,52
3
92.61 =1.1
86.56 +5.0
99.66 20.4
12
99.10 +1.0
4
93.16 =0.7
12
84.85 #6.5
99.80 =0.2
13
99.56 #0.6
5
87.17 #4.8
13
87.42 #6.4
99.82 20.3
98.98 #1.1
6
87.69 +2,9
87.73 +6.0
99, 99.98 =0.0
99.19
#6.1
99.88 10.1
16
99.80 #0.4
8
86.32 16.1
92.10 +4.9
4.4 Further Analysis
Disentanglement Robustness Analysis. To evaluate the robustness of our action disentanglement, analyze the a, accuracy of videos with different numbers of ground-truth labels (denoted as label nums Table ([7) when exposed to varying numbers of present labels the SAP, under both distractor-injectee and non-distractor settings. As shown in Table 7 in the distractor-injected (SAP contains the labels not present video), we observe a non non-monotonic trend: accuracy drops when the number of present labels increases from 5 12, but rises again as present labels dominate the input. We attribute the initial drop to the increasing complexity of the prediction task due to more competing present labels under noisy input. The subsequent performance increase attributed to reduced noise as more present labels are included. Despite the precision varying between 85% and 100% under distractor-injected,, our model demonstrates robust performance across different input conditions. Even in the worst-case scenario, where the precision is at its lowest (85%), the model still maintains a relatively high accuracy. This indicates its ability handle incomplete or noisy data effectively. Moreover, in the non non-distractor-injetee case (SAPs only contain labels present in video), as shown in T the model achieves nearly perfect accuracy, reaching close to 100%. This shows that in ideal conditions, the model is able to fully exploit the available information. The robustness in noisy conditions and strong performance on clean data indicate that our model generalizes well and benefits from mutual reinforcement between the two settings.
Temporal Action Localization. Although our method does not explicitly design localization module, the disentangled representation naturally provides cues for action localization. Specifically, within our readout function, aggregate temporal features using learned frame-level weights s. These weights reflect the relevance of each frame to the target action, and can thus be interpreted frame-level localization scores. By normalizing these weights, we obtain a per-frame prediction that serves as our localization output. To obtain frame-level ground truth for localization, we project the action segment annotations onto frames: frames within the annotated action duration are labeled as positive (1), and the rest negative (0). We binarize the normalized weights using threshold 0, assigning 1 to frames with using frame-level mAp based a,, As shown in Table 8 we observe that the variation in average precision across different values is larger than the variation across different IoU thresholds, and the gap between tistractor-injecte and non con-distractor-injeee settings narrows 0 increases. This
and otherwise, and evaluate the resulting predictions

---

(a)
(b)
Figure 4: t-SNE visualization of disentangled action features across 157 classes on Charades 40 (a)
Training set. (b) Test set.
Table 8: Distractor-injected SAP
Table 9: Non-disttacttorinjeceee SAP
IoU threshold
IoU threshold
0.5
Avg.
0.5
Avg.
0.2
77.97
43.68
26.08
49.24
0.2
85.56
57.44
40.18
61.06
0
0.5
64.25
33.70
20.78
39.57
0
0.5
74.44
43.43
27.75
48.53
0.7
53.80
22.98
12.99
29.92
0.7
59.27
24.26
13.14
32.23
Avg.
65.34
33.45
19.95
Avg.
73.09
41.71
27.03
indicates that the introduced noise (i.e., absent labels) mainly adds irrelevant information without significantly disrupting the separation of true actions, demonstrating the robustness of our method. In the con-disractoo-injecte setting, the model achieves notably better localization performance, confirming that the high recognition accuracy is not due to shortcut learning, rather due effective utilization of SAP information for representation disentanglement under clean inputs.
4.5 Visualization
We computed the inter-class and intra-class distances after applying t-SNE 32 for dimensionality reduction on Charades [40], with the visualization results shown in Figure 4 Specifically, for each video with labels, we disentangle L action features, covering a total of 157 action classes. We further calculate the inter-class and intra-class distances on both the training and test sets. For the training set, the intra-class and inter-class distances are 2.08 1.57 and 58.5 58 + 27.99, respectively, while for the test set they are 1.79 + 1.45 and 74.94 + 36.98. Notably, the intra-class distances of disentangled features are mostly within 1, whereas the inter-class distances remain substantially larger. This demonstrates that our feature discntanglement effectively increases the separability between action categories while enhancing the compactness of features within the same category, thereby providing more discriminative representations for downstream action recognition and understanding. provide more visualization results in See
5 Conclusion
In this paper, we propose Prompt-guided Representation Disentanglement for Action Recognition (ProDA), which is capable of disentangling any complete action from Spatio-temporal Scene Graphs (SSGs). ProDA leverages SSGs and introduces Dynamic Prompt Module (DPM) to guide Graph Parsing Neural Network (GPNN) in extracting action-specific representations. We further design video-adapted GPNN that aggregates information using dynamic weights. To enhance performance, we introduce VGNorm and AD Loss, which ensure the spatio-temporal consistency across SSGs of different videos and the completeness of feature disentanglement, respectively. Experimental results demonstrate the superiority of our method.
10

---

Acknowledgments and Disclosure of Funding
This work supported by grants from the Natural Science Foundation of Shanxi Province (2024ICICON-60) Science and Technology Commission of Shanghai Municipality (NO.24511 106900), Key R&D Program of Zhejjang (20248SYS0091).
References
[1] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv:1607.00 064.00000000. 2016.
arXiv preprint
[2] Hyojin Bahng, Ali Jahanian, Swami Sankaranarayanan, and Phillip Isola. Exploring visual prompts for adapting large-scale models. arXiv preprint arXiv:2208.17774 2022.
[3] Tianle Cai, Shengjie Luo, Keyulu Xu, Di He, Tie-yan Liu, and Liwei Wang. Graphnorm: principled approach to accelerating graph neural network training. In International Conference on Machine Learning. pages 1204-1215. PMLR, 2021.
[4] Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? new model and the kinetics dataset. In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2299-6308 2017.
[5] Y. Cong. W. Liao, H. Ackermann, et al. Spatial-temporal transformer for dynamic seene graph generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 16372- 16382, 2021.
[6] Emily L Denton et al. Unsupervised learning of disentangled representations from video. Advances in neural information processing systems, 30, 2017.
[7] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenbom, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010.1 1190,1 2020.
[8] Taoran Fang. Yunchao Zhang, Yang Yang. Chumping Wang. and Lei Chen, Universal prompt tuning for graph neural networks. Advances Neural Information Processing Systems, 36:52464-52489, 2023.
[9] Christoph Feichtenhofer, Axel Pinz, and Andrew Zisserman. Convolutional two-stream network fusion for video action recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1933-1941, 2016.
[10] Christoph Feichtenhofer, Axel Pinz, and Richard P Wildes. Spatiotemporal multiplier networks for vidco action recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4768-4777, 2017.
[11] Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, al. Slowfast networks for video recognition. In Proceedings of the IEFF/CVF International Conference on Computer Vision, pages 6202-6211, 2019.
[12] Lianqiang Gan, Junyu Lai, Jingze Ju, Lianli Gao, and Yi Bin. Dfdnet: Disentangling and filtering dynamics for enhanced video prediction. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 3059-3067, 2025.
[13] Rohit Girdhar, Joao Carreira, Carl Doersch, Andrew Zisserman. Video action transformer network. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages 244-253, 2019.
[14] Arthur Gretton, Alexander Smola. Olivier Bousquet, Ralf Herbrich, Andrei Belitski, Mark Augath, Yusuke Murayama, Jon Pauls, Bernhard Schölkopf. and Nikos Logothetis. Kernel constrained covariance for dependence measurement. In International Workshop on Artificial Intelligence and Statisticx, pages 112-119. PMLR, 2005.
[15] Mingfei Han, David Junhao Zhang, Yali Wang, Rui Yan, Lina Yao, Xiaojun Chang, and Yu Qiao. Dual-ai: Dual-path actor interaction learning group activity recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1990-29991 2022.
[16] Tal Hassner. A critical review of action recognition benchmarks. In proceedings of the IEEE conference computer vision and pattern recognition workshops, pages 245 250-250, 2013.
11

---

[17] Jun- Jun-Ting Hsieh, Bingbin Liu, De-An Huang, Li F Fei-Fei, and Juan Carlos Niebles. Learning to decompoee and disentangle representations for video prediction. Advances in neural information processing xystems, 31, 2018.
[18] Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pages 448-456. pmir, 2015.
[19] Ji Ji, Rishabh Desai, and Juan Carlos Niebles. Detecting human-object relationships in videos. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8106-8116, 2021.
[20] Jingwei Ji. Ranjay Krishna, Li Fei-Fei, and Juan Carlos Niebles. Action genome: Actions as compositions of spatio-temporal scene graphs. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10236-10247, 2020.
[21] Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser- Nam Lim. Visual prompt tuning. In European conference computer vision, pages 709-727. Springer, 2022.
[22] Wentao Jiang, Si Liu, Chen Gao, Jie Cao, Ran He, Jiashi Feng, and Shuicheng Yan. Psgan: Pose and expression robust spatial-aware gan for customizable makeup transfer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5194-5202, 2020.
[23] Kumara Kahatapitiya, Anurag Arua,, Arsha Nagrani, and Michael S Ryoo. Vietr: conditiodi text representations for activity recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18547-18558, 2024.
[24] Haoyang Li, Xin Wang, Ziwei Zhang, Zehuan Yuan, Hang Li, and Wenwu Zhu. Disentangled contrastive learning graphs. Advances in Neural Information Processing Systems, 34:21872-21888 2021.
[25] Haoyang Li, Ziwei Zhang, Xin Wang, and Wenwu Zhu. Disentangled graph contrastive learning with independence promotion. IEFE Transactions on Knowledge and Data Engineering. 35(8):7856-7869, 2022.
[26] Haoyang Li, Xin Wang, Zeyang Zhang, Haibo Chen, Ziwei Zhang, and Wenwu Zhu. Disentangled graph self- supervieed learning for -fitt--diitiittton generalization. In Forty-first International Conference on Machine Learning, 2024.
[27] Yanghao Li. Chao- Yuan Wu, Haoqi Fan, Karttikeya Mangalam, Bo Xiong, Jitendra Malik, and Christoph Feichtenhofer. Mvitv2: Improved multiscale vision transformers for classification and detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4804-4814 4814,, 2022.
[28] Yixuan Li, Lei Chen, Runyu He, Zhenzhi Wang, Gangshan Wu, and Limin Wang. Multisports: A multi- person video dataset of spatio-temporally localized sports actions. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 13536-13545, 2021.
[29] Zhuolong Li, Xingao Li, Changxing Ding, and Xiangmin Xu. Disentangled pre-training for human- object interaction detection. Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 28191-28201, 2024.
[30] Ze Liu, Jia Ning, Yue Cao, Yixuan Wei, Zheng Zhang, Stephen Lin, and Han Hu. Vidco swin transformer. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3202-321 11, 2022.
[31] Yihong Ma. Ning Yan. Jiayu Li. Masood Mortazavi, and Nitesh V Chawla. Hetgpt: Harnessing the power of prompt tuning in pre- trained heterogeneous graph neural networks. In Proceedings of the ACM Web Conference 2024, pages 1015-1023, 2024.
[32] Laurens van der Maaten and Geoffrey Hinton. Visualizing data using I-sne. Journal of machine learning research, 9(ov))2559-2665 2008
[33] Joanna Materzynska, Tete Xiao, Roei Herzig, Huijuan Xu, Xiaolong Wang, and Trevor Darrell, Something- clse: Compositional action recognition with spatial-temporal interaction networks. In Proceedings of the JEEE/CVF conference on computer vision and pattern recognition, pages 1049-1059, 2020.
[34] Yujie Mo, Yajie Lei, Jialie Shen, Xiaoshuang Shi, Heng Tao Shen, and Xiaofeng Zhu. Disentangled multiplex graph representation learning. In International conference on machine learning pages 24983- 25005. PMLR, 2023.
12

---

[35] Yangjun Ou, Li Mi, and Zhenzhong Chen. Object-relation reasoning graph for action recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20133- 20142, 2022.
[36] Junting Pan, Siyu Chen, Mike Zheng Shou, et al. Actor-context-actorr relation network for spatio-temporal action localization. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recogni- tion, pages 464-474, 2021.
[37] Siyuan Qi. Wenguan Wang. Baoxiong Jia, Jianbing Shen, and Song-Chun Zhu. Learning human-object interactions by graph parsing neural networks. Proceedings of the European conference on computer vision (ECCV), pages 401-417,22 2018.
[38] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transfcrable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763. PmL.R, 2021.
[39] Zhengxiang Shi and Aldo Lipani. Dept: Decomposed prompt tuning for parameter-efficient fine-tuning. arXiv preprint arXiv:2309.cm 05173, 2023.
[40] Gunnar A Sigurdsson, Gül Varol, Xiaolong Wang, Ali Farhadi, Ivan I Laptev, and Abhinav Gupta. Hollywood in homes: Crowdsourcing data collection for activity understanding. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 1-14, 2016, Proceedings, Part I 14, pages 510-526. Springer, 2016.
[41] Karen Simonyan and Andrew Zisserman. Two-stream convolutional networks for action recognition in videos. Advances in neural information processing systems, 27, 2014.
[42] Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri, Learning spatiotemporal features with 3d convolutional networks. In Proceedings of the IEEE international conference computer vision, pages 4489-4497, 2015.
[43] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones. Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances neural information processing systems, 30, 2017.
[44] Heng Wang and Cordelia Schmid, Action recognition with improved trajectorics. In Proceedings of the IEFE international conference on computer vision, pages 3551-3558, 2013.
[45] Junke Wang, Dongdong Chen, Chong Luo, Bo He, Lu Yuan, Zuxuan Wu, and Yu-Gang Jiang. Omnivid: A generative framework for universal video understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19209-18222 2024.
[46] Ning Wang, Guangming Zhu, Hongsheng Li, Mingtao Feng, Xia Zhao, Lan Ni, Peiyi Shen, Lin Mci, and Liang Zhang. Exploring spatio-tempora temporal graph convolution for video-based human objeett interaction recognition. IEEE Transactions on Circuits and Systems for Video Technology, 33(10):5814-5555 14-7727, 2023.
[47] Ning Wang. Guangming Zhu, HS Li, Liang Zhang, Syed Afaq Ali Shah, and Mohammed Bennamoun. Language model guided interpretable video action reasoning. Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 18878-18887, 2024.
[48] Xiaolong Wang and Abhinav Gupta. Videos as space-time region graphs. In Proceedings of the European conference on computer vision (ECCV), pages 399-417, 2018.
[49] Yunbo Wang. Mingsheng Long, Jianmin Wang, and Philip S Yu. Spatiotemporal pyramid network for video action recognition. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 1529-1538, 2017.
[50] T. Wu, R. Нe. G. Wu, al. Sportshhi: dataset human-human interaction detection in sports videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18537-18546, 2024.
[51] Saining Xie, Chen Sun, Jonathan Huang, Zhuowen Tu, and Kevin Murphy. Rethinking spatiotemporal feature learning: Speed-accuracy trade-offs in vidco classification. In Proceedings of the European conference compuder vision (ECCV), pages 305-321, 2018.
[52] Boqiang Zhang, Hongtao Xie, Zuan Gao, and Yuxin Wang. Choose what you need: Disentangled representation learning for scene text recognition removal and editing. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 28358 28366, 2024.
13

---

[53] Ce Zhang. Changcheng Fu, Shijie Wang, Nakul Agarwal, Kwonjoon Lee. Chiho Choi, and Chen Sun. Object-centric video representation for long-term action anticipation. In Proceedings the IELE/CVF Winter Conference on Applications of Computer Vision, pages 6751-6761, 2024.
[54] Jingran Zhang. Fumin Shen, Xing Xu, and Heng Tao Shen. Temporal reasoning graph for activity recognition. IEEE Transactions on Image Processing, 29:5491-5506, 2020.
[55] Yubo Zhang, Pavel Tokmakov, Martial Hebert, and Cordelia Schmid. structured model for action detection. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9975 9984, 2019.
[56] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, Ziwei Liu. Conditional prompt learning vision- language models. In Proceedings of the IEEE/CVF conference on computer vision and puttern recognition, pages 16816-16825, 2022.
[57] Xingyi Zhou, Anurag Amab, Chen Sun, and Cordelia Schmid. How can objects help action recognition? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2353-2251 3362. 2023.
[58] Yizhe Zhu, Martin Renqiang Min, Asim Kadav, and Hans Peter Graf. S3vae: Self- sequential for representation disentanglement and data generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6538 6547, 2020.
[59] Tao Zhuo, Zhiyong Cheng, Peng Zhang, Yongkang Wong, and Mohan Kankanhalli. Explainable vidco action reasoning prior knowledge and state transitions. In Proceedings of the 27th aem international conference on multimedia, pages 221-529, 2019.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction accurately describe the proposed framework Prompt-guided Disentangled Representation for Action Recognition for multi-action scene, and achieve the state-of-the-art results multi-label benchmark.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We provide limitations in the appendix.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
15

---

Answer: [NA]
Justification:
No theoretical results are provided in our method.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Yes, we use publicly available datasets, and detailed code along with implemen- tation details will be provided in the supplementary material to ensure full reproducibility of the main experimental results and support the claims and conclusions of the pap Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
16

---

reproducible, they Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Detailed code for reproducing the main experimental results are provided in the supplementary material. We also use publicly available datasets.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.c cc/
public/guides/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/
Cooee//dddeeebmimiia ssi onPolicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [No]
Justification: The full details can be provided either with the code, in supplemental material.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: We do not report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
17

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative еттоr rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [No]
Justification: Since some of the compared methods do not provide complete computational resource information (and some not release their code), it is challenging for us to perform a fair comparison regarding computational cost.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics http:////eeaaapp.ccccc//bb////ba
Answer: [Yes]
Justification:
Yes, followed.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [No]
We propose a prompt-guide disentangled representation for action recognition, which offers novel way to disentangle actions. We believe it is natural for peers to build upon our work to improve it or apply it to other directions, such as temporal action localization. Therefore, we did not dedicate a separate section for discussion the paper.
Justification:
Guidelines:
The answer NA means that there is no societal impact of the work performed.
18

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
Our paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [NA]
Justification: Our paper does not use existing assets.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
19

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssi
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: Our paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: Our paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
20

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM for what should or should not be described.
21

---

A
Video Encoder
We adopt Dual-Al [3] as our video encoder, which a two-stream architecture composed of interleaved temporal and spatial Transformers [7]. This design facilitates effective spatio-temporal information exchange through alternate modeling along temporal and spatial dimensions. On top of the original architecture, branches to enhance the flow of information across both dimensions. The overall architecture is illustrated in Figure [ After the video encoder, an Multi-Layer Perceptron (MLP) fuses the two branches, and a Feed Forward Network (FFN) [43] further integrates the features to produce the input F!. for Video Graph Parsing Neural Network (VGPNN).
further introduce additional spatial-to-spatill and temporal-to-temppral
Both the Spatial Transformer and the Temporal Transformer adopt the standard self-attention encoder architecture [4 (as shown in Figure O), with the primary difference lying in the dimension along which the attention is applied in spatial or temporal dimension, as shown in Figure 7
Obyec
formformer
Spatial
ML.P
Transforner
Temporal
Forward
MLP
Feed
Tranformer
Temporal
Transformer
tempoall
ML.P
Spatial
Scene Graph
Scene Graph
Input
F
Figure 5: Our video encoder builds upon Dual-AI [5 by introducing additional spatial -pattill and temporal-tempont streams.
Add & LayerNorm
Add LayerNorm
Self- Aaf-AAtan Attaattoon
Self- Self-Attrrri Attartioo
to +
Multi-Head Allention
Forward
Block
Block
Feed
+o
Spatial Transformer
Temporal Transformer
Figure 7: Spatial Transformer and Temporal Trans-
Figure 6: Architecture of self-attention en- former coder.
B
Video Graph Parsing Neural Network
B.1
Formulation
In SSG, each frame of a video processed into graph, where nodes represent the objects present in that frame, and edges represent the semantic relationships between these objects (e.g., spatial or interaction relations). Formally, let 9 (V, &, R) denote the complete SSG constructed from the video, where V is the set of all object nodes across all frames, E is the set of all directed edges, and is the of all relation types. Each node E V is uniquely indexed from {1..... [V]]. Each edge (v, W. r) E V V represents directed semantic relation of type Let g. (V,, &,, R.) denote the scene graph the i-th frame, where C V, E, S & and R; R denote the frame-specific object nodes, their edges and relations. We denote vf as the j-th object node in frame i, and eith the edge from vę to of with relation type r. R... The node features be denoted Faode (st f?..... fM E NxDD where f B represents the feature of node v! the j-th frame. Here, N, M, and denote the number of frames in the video, the number of nodes per frame, and the dimensionality of each node feature, respectively.
R. from node to node
22

---

It is important to note that the number of nodes in each frame is the same. The i-th node in every frame corresponds the same object, though its visual features may vary across frames. This ensures that each node represents the same object across frames, even though its appearance changes. However, not every object appears in every frame, leading to padding where necessary.
Link. For edge ei*A: we have N triples across frames, denoted as {(v], vf. r!.")...., (uly. ox x }} To model the transition of object interactions over time, we first extract interaction features from these edge triples. Specifically. we concatenate the components of each edge triple and feed them into edge function Edge(-) (implemented using Multi-Layer Perceptron, MIPs; unless otherwise specified, all functions in subsequent modules are implemented as MI.s) to obtain Trame-wise relation features fi.k = [ff f2 ...., JN }. We then apply a self-attention encoder [43 to model the dependencies and transitions among these interaction features across frames to obtain interaction feature Filk {Fi Fijh ...... Ria} follow:
fi.ky
Filk SelfAttEncoder
Edge([u] (Edgee 11 11r1111)............. (Lew ox N 1)
(9)
relation features f2.A
where [-] represent concatenation, and SelfAttEncoder (-) is the self-attention encoder with L standard self-attention encoder blocks (as shown in Figure
Message & Update. To facilitate better disentanglement, we employ dynamic weights to modulate the information propagated through different edges and nodes. This not only enables selective information flow, but also provides interpretability by highlighting the importance of each interaction. We through interaction feature Fiik obtain weight feature F) as follows:
Fi.k oW" FTih))
(10)
where m denotes a linear transformation projecting to a scalar. We then combine the interaction features and node features to compute the messages F% to be passed to node v vg.......... wx
in each frame, which are dynamically aggregated using the weight features F as follows:
FJ m
Σ k=
(Fi Msg((1) F).)))
(11)
where denotes the number of neighbors connected to node v and Msg(-) represents the message function. After computing the message features Fi we use aggregation function incorporate them into the updated representation of node Uj. as follows:
Agg(v @ Flh
(12)
where v Agg(-). and + denote updated node representation, the aggregation function, and element- wise addition, respectively. 3 replaces v in following processing.
Compared previous work Graph Parsing Neural Network (GPNN) B7. VPGNN introduces
stronger temporal modeling capabilities by modeling edges over time, rather than just generating aggregation weights for edges. VGPNN updates node features each layer, while GPNN primarily updates edges and only updates nodes in the final layer. Additionally, VGPNN retains the full SSG across frames, unlike GPNN, which only preserves nodes from single frame. This results VGPNN offering better interpretability and generalization.
C
Action Specification
C.1 Design
We assume that the label length of a dataset is 5. For a video labeled with classes (0, 1}, its multi-hot representation is:
1, 0, 0]
(13)
Then, there will be three Specified Action Prompts (SAPs), corresponding to (0, 1). (0), and (1) respectively (As shown in Eq. 14)
[0,1] + [,,000,0,0]
(0) + [1,0,000,0]
(14)
[0,10000,
23

---

However, using only the SAP may lead the model to take shorteuts, focusing solely on the SAP rather than the video content. Therefore, we augment the SAP with distractor classes that are not associated with the video. In this case, the distractor set {2, 3, 4). Assuming K=3, we take the second example Equation 14 (i.e., with label (0) tto illustrate: will generate three distractor-injected SAPs (As
shown in Eq. 15
O, 1,
[1,0, 1,0,
(15)
[1,0,0,1,1]
Similarly, taking the first example Eq 14 (i.e., with label {0, 1)), with label, will also generate three distractor-injected SAPs (As shown Eq I6).
[L, 1, 1,
11,10,1100,
(16)
11,1,0,0,11 1]
In both examples, the number of ones in the multi-hot vector
We divide the features into two parts: one corresponding to the specified action and the other to the remaining actions. If we feed the SAP into the part that is intended for unrelated actions, will lead to information leakage, since the SAP contains the ground-trume labels. To avoid this, we instead feed the full label space of the dataset. Specifically, we obtain the Unspecified Action Prompts (UAP) by taking the complement of the SAP with respect to the entire label space, such that the combination of SAP and UAP covers action categories the dataset.
Taking Eq. 15 as an example, its SAP is given by:
[0, 1, 0, 1]. 00 0,,10,1,00
(17)
00,1,
For Eq. 16 its SAP is given by:
[0, 1,
00,0, 1,0,1
(18)
0, 1,
C.2
Construction
The AS (SAP and UAP pair) consists of two types: distractor-injected and non-distractor-ineeet The iistractor-injeeted AS includes a combination present labels (i.e., labels of some actions in groundtruth labels) and absent labels (i.e., labels of some actions absent in the groundtruth labels), while non-distractor-iijeetee only contains present labels.
For a video with the ground truth label set (Putting a pillow somewhere, Taking a pillow from somewhere, Throwing pillow somewhere, Holding pillow, Tidying up a blanket/s) corresponding to label indices 177,79.00,76,77 The remaining labels, i.e., other labels from the full set {0, 1..... 156} excluding the ground truth labels. Absent labels are sampled from remaining labels.
and presents labels are sample from this ground truth label set.
Under the non-distractorrinjeer setting, we generate SAPs:
[77]
[80, 76],
[77, 80, 75).
(19)
779, 77, 76, 75),
[80, 77, 76, 75, 79)
Fach SAP contains only present labels, with ground truth labels randomly sampled from the ground truth label set. To prevent information leakage, use all labels outside each SAP as the

---

corresponding UAP, thereby disentangling the rest of actions. Consequently, for the 5 SAPs, we generate 5 corresponding UAPs.
present labels: 880,76,75,79, 0, I...., 156).
156 labels
present labels:
77,,9,75,, 0, 1,..., 156
155 labels
2 present labels: 79,76, 0, 156
(20)
154
present labels: (80, 0, ... 156
153 labels
present labels: [0, 1,..,, 156
152 labels
Under the iisteee setting, we fix the length of each SAP to 16, and generate 5 + 1 6 SAPs:
present labels:
(2, 112,
24, 125, 145, 110, 93, 138, 129, 44, 124, 130, 7,5, 0).
present labels: [76, 143, 19, 104, 112, 12, 146, 56, 136, 116,9, 9,24, 125].
present labels: [77, 76, 140, 111, 23, 18, 155, 136, 133, 112, 127, 17,2],
(21)
3 present labels: [76, 79, 75, 47, 91,, 66, 141,9,27,52, 1332, 54],
present labels: [76, 79, 75, 77,56, 88, 120, 22, 128, 62, 52, 105, 142, 124, 153, 35]
5 present labels: 880 77, 76, 75, 79, 17,51, 109, 143, 103, 57, 15, 152, 130, 125, 47]
Each SAP contains both present and absent labels, with 0 to 5 ground truth labels randomly sampled from the original ground truth label set, and the rest 16 to labels sampled from the remaining labels. To prevent information leakage, use all labels other than those in each SAP the corresponding UAP, thereby disentangling the other actions. Each UAP has length of 141. Consequently, for the five SAPs, we generate 6 corresponding UAPs, each of length 141.
present labels: 80,76,77,75,,99, 1,2,
156
present labels:
880,77,75,79, 0, 1. 156)
141 labels
present labels:
880,79,75, 0, 1, 156)
141 labels
(22)
2 present labels:
[80,77, 0, 2, 156)
labels
present labels: 800 0, 156
141 labels
present labels: 00, 1 1,.., 156
141 labels
Finally, for video with 5 labels, generate 2 5 +-=1= 11 ASs (11 SAP and UAP pairs).
25

---

Table 10: This table outlines the division of the Charades dataset into five subsets, ensuring that there is no overlap between the scenes used for training and testing
Subdataset
Training Scene
Test Scene
Stairs, Laundry room, Office.
Living room, room,Closet, Kitchen, Bathroom.
Scenariol
Hallway, Bedroum. Diming tntyyynyry
Gurage.Recraatinn Recreation room, Busement, Other
Laundry womm,athroom Saatmm.aannnniiiiia Closet, Entryway. Recreation room, Garmee, Other
Bedroom, Living room, Kitchen,Hort Office,
Scenario2
Hallway,Stairs, Biallayy, Dining room
Stairs, Laundry room, Bedroom, Basement, Bathroom, Entryway, Recreaatin room,
Living room, roose,loset, Kitchen, Barngg,,alllay,Paaaee Dining room
MinnCCosettKKtchennn,,o Home Office,
Scenario3
Fiichen,Stairs, Laundry room, Home Office, Bedroom, Bathroom, Pantry, room
Living room, room,Closet Garaee Garage,Hallway Recreation room, Noom,Entrywar Basemma....ttt... Other
Scenario4
Kitchen, Laundry room, Hool,ay Basement,
Bedroom, Office, Bathroom, Baathroo,,Garage, Recreatecr room,Entry way.Pantry
ScenarioS
Dining Living room, Clon,C,l
Table 11: Domain shift experiment on the Charades dataset
Method
mAP (%)
STIGPN 46
54.1
LalAR
44
57.2
Ours
63.0
D Details and More Results for Our Experiment
D.I Additional Experiment
We conduct domain shift experiments the Charades dataset to evaluate the generalization ability of our method. Specifically, we follow the experimental setup of [47). splitting Charades into five disjoint subsets with different action groups. The details are shown in Table I0
As shown Table II our method achieves the best performance under domain shift settings, in- dicating strong generalization ability across different scenarios. We attribute this to the effective disentanglement of features, which allows the model to better capture action-relevant information, thereby improving action understanding and enhancing generalization performance in unseen envi- ronments.
D.2
More Details
For each video, we uniformly sample 16 frames input. We adopt two-stage training strategy for all experiments. The first stage (pre-train) runs for epochs with a learning rate of 2e-4, aiming to learn the representations for an and a,, In the second stage (post-train), we load the checkpoint from the epoch with the best performance on au and ag in the pre-training stage, and only train the classifier for a,. This stage is also trained for epochs with learning rate of le-4. We report the west-performing epoch from the second stage. The detailed results are shown in Table 12 [3 Except for our method, other approaches generally show a certain degree of performance improvement across different settings. Notably, our method consistently achieves the best performance in all scenarios, demonstrating its robustness and superior generalization capability.
E
Visualization
Figure 3 shows the action disentanglement localization results for two different videos (with 3 and 4 action labels respectively). Both videos contain concurrent or individual interactions. Figure 9 illustrates the process by which our method disentangles the specified actions from the concurrent interactions.
26

---

Table 13: Ablation study of loss function on Charades dataset. denotes using ProDA with only the action classification loss.
Table 12: Ablation of normalization
Post-train o,mAP (%)
Pre-train a,mAP (%)
norm lype
Cuis w/o margin
Post-train 4,mAP (%)
Pre-train 0,mAAP (%)
frex
Cais
LayerNorm
68. 12
65.44
w/o margin
BatchNorm 18
65,39
63.05
67.45
65.83
GraphNorm 3
68.61
65.32
68.43
18:27
67.78
66.92
VGNorm
71.10
71.32
71.10
71.32
06
Action Holding a vacuum
Action: Turning off a light
Figure Frame-level weights s predicted by the readout function for different non iiistractor-injected SAPs within the same video. The heatmaps (colored bars) visualize the predicted weights, reflecting the relevance of each frame to the target action. The red bars indicate the ground-truth action segments, while the green bars denote our localization results obtained by thresholding at 0.7.
In Figure 10 we show more complex example from video with 15 labels, containing multiple concurrent interactions. In this example, the person interacts simultaneously with a table, chair, cup, book, and food. Using prompts, we disentangle the interaction between the person and the shoe from these overlapping interactions. Additionally, the subgraphs relevant to the specified action are successfully parsed from the full SSG. In Figure II present example from video with 10 labels, which also contains multiple concurrent interactions. In this example, the person interacts simultancously with sofa, blanket, TV, and food. Using prompts, disentangle the interaction between the person and the shoe from these overlapping interactions. Additionally, the subgraphs relevant the specified action successfully parsed from the full SSG.
F
Limitation
Our method operates solely on visual features and the derived PentiolTempora Scene Graphs (SSGs), without incorporating other modalities such audio, text, or human pose. This reliance on single modality may limit the flexibility and robustness of the model, especially in scenarios where multimodal cues are essential for disambiguating actions. future work, we plan to extend our framework to integrate multimodal information, enabling more comprehensive and robust action disentanglement complex environments.
27

---

Parsing
Windee
Faree
Parsing
Yacmum
VACKARI
<eeson, indow> <Person, V Vacuum>
Interactions:
<Person, W Indow>
Inter actions:
Interactions: <Person, V Vacuum>>
Interactions
<Person, V Vaccum>>
4erson, acuumm
(a) Holding a V Vacuum
Parsing
Bag
Blg
Parson
Persen
Peens
Parsing
Swisch
Interactions <Peson,Bagg> 4 PeeroonSwiichh>>
Interactions: Bag>
Interactions <Person,Switch>
Interactions:
<Person,Switche
<Person,Switche
(a) Tum off Light
Figure 9: Key parsing steps for the two examples presented in Figure For each case, the specified action is disentangled from multiple co-occurring interactions. The boxes indicate the objects retained by VGPNN during the parsing process. Initially, the scene graph is fully connected with all nodes; after parsing, only subset of relevant nodes remains. Below each image is the corresponding scene graph of that frame
28

---

Adion Taking cupiglass/boitle from somewhere
Paraed Result,
Spacified Action Suitraaphs
Firee
Hodding sume food. Sitting at table
Opentng beek
Bolding a oupiglazcbitll of sooething arking papertaatebesk Taldng sanewbere
Tabing crpodanchettle from semmebere
Putling cupighen bottle uasneelave
Warking
Semrene helding paperhotebaok
atcciegyRoddieg tokkng book
Sittagg
Sumsun nallng acnathing
Kpet
Labrds
Labels
Figure 10: This is the parsing process of a 15 labels video. We disentangle a single interaction from series of complex interactions the scene by using prompts. The top part of the image shows our Temporal Action Localization results.
Action: Putting shoe/shoes
Dew
RasseRReall Resull Specified Action Subgraphs
Lyiag se sefaitouch
Favting a wenheich
Semsone calling somsthing
showrdhame
Watcaing telettiin
Hobling blunket
Itoktng some feed
Seascos sitanding fron sumeelare
Sitting en selalonrch Souppling blankett
Figure 11: This is the parsing process of a 10 labels video. We disentangle a single interaction from series of complex interactions the scene by using prompts. The top part of the image shows our Temporal Action Localization results.
29


Reviews:
[{"review_id": "wOtz4R2ZM9", "reviewer": "Reviewer_MhYr", "summary": "This paper addresses the challenge of accurately recognizing specific actions in complex, multi-action video scenes, where conventional methods typically extract unified features and struggle to capture action-specific semantics and inter-object interactions. To tackle this, the authors propose a novel framework called ProDA (Prompt-guided Disentangled Representation for Action Recognition), which enables the model to disentangle and analyze any specified action within a scene by leveraging Spatio-temporal Scene Graphs (SSGs) and prompt-based conditioning. Specifically, ProDA introduces a Dynamic Prompt Module (DPM) to generate action-aware prompts, a customized Video Graph Parsing Neural Network (VGPNN) to extract action-relevant subgraphs, and an Action Disentanglement Loss that guides the network to learn semantically meaningful and reconstructable representations. Without requiring explicit action localization labels, ProDA achieves state-of-the-art performance on multi-label action recognition benchmarks while maintaining strong interpretability and localization capabilities.", "questions": "1. Can the authors provide inference speed or computational cost analysis?\n2. Could the authors include a qualitative visualization (e.g., t-SNE) comparing disentangled vs. unified features?\n3. Why is the evaluation limited to a single dataset? \n4. How sensitive is ProDA to errors in SSG construction?\n\nIf these issues are properly addressed, I would be happy to raise my evaluation score.", "limitations": "Yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The paper is motivated by a clear and relevant problem — the difficulty of recognizing specific actions in multi-action scenes using unified features. \n2. The writing is well-structured and easy to follow. \n3. The proposed method does not require action localization supervision.\n\nWeaknesses:\n1. The paper lacks analysis of the model's inference efficiency, which is important given the complexity of the proposed pipeline. \n2. While the motivation for action-centric feature disentanglement is sound, the paper does not provide visual evidence (e.g., t-SNE) to demonstrate the effectiveness of the disentangled representations compared to conventional unified features. \n3. Evaluation is limited to a single dataset, raising concerns about generalizability. \n4. Additionally, the potential impact of SSG quality on model performance is not analyzed or discussed, which could affect robustness in real-world applications.", "was_revised": true, "final_justification": "The authors have added experiments and addressed some of my concerns, but there is still room for improvement in terms of experiment completeness and visualization (I understand that it may not be convenient to add images during the rebuttal period). I maintain my score, but this paper can indeed be accepted.", "rebuttal": "We thank the reviewer for the positive comments on our motivation, writing, and the design of our method without action localization supervision. The reviewer also raised concerns, which we address below.\n\n**Q1:** Can the authors provide inference speed or computational cost analysis?\n\n**A1:** In addition to the dataset used in our main experiments, we conduct our experiments on SportsHHI , a recent benchmark focusing on human-human interaction in sports videos. We conducted an analysis of inference speed and computational cost on the SportsHHI dataset. Since the architectures used in prior works vary significantly, direct comparison is not straightforward. In our case, we integrated the disentanglement mechanism into a unified baseline architecture, which enables a fair comparison by isolating the performance improvements and overhead introduced specifically by the disentanglement module.\nWe conduct two experiments for fair and thorough evaluation:\n- Module Integration: We build upon the official SportsHHI  baseline (implemented with MMAction2) by integrating our designed modules, including two Dynamic Prompt Modules (DPM)—one for the Unspecified Action Prompt (UAP) and one for the Specified Action Prompt (SAP)—as well as the proposed Action Disentanglement Loss (AD Loss).\n- Parameter-matched Baseline: To ensure a fair comparison, we expand the baseline model by increasing its parameters to match the model size of our method, without adding our modules.\n\nIn the reported parameter count, we excluded the parametersof the Reconstruction Network, as it is not involved during inference. **For fair comparison, we reimplemented a version of the original model with larger parameter capacity, while keeping the architecture and training configuration consistent**\n\nAs results shown in Table 1, our method introduces 10.8% more parameters than the baseline, but only increases inference time by 8.89%, indicating that the additional modules do not significantly compromise efficiency. More importantly, our method achieves a promising improvement in accuracy (+2.86 mAP). Furthermore, under our reimplemented version whose parameters are compatible with our method, our method still outperforms the baseline (+3.39 mAP),  demonstrating that the performance gain mainly comes from our proposed disentanglement mechanism rather than model size alone.\n\nSportsHHI  provides fine-grained annotations of interactions between players. SportsHHI  contains 34 high-level interaction classes from basketball and volleyball sports. 118,075 human bounding boxes and 50,649 interaction instances are annotated on 11,398 keyframes. Compared to conventional action datasets, SportsHHI offers richer relational labels and poses greater challenges due to dense, multi-person interactions\n\n**Table 1:**  Human-Human Interaction Classification results in SportsHHI.SportsHHI\\* is our reimplementation based on the official SportsHHI code, with minor modifications (e.g., parameter size) to match the capacity of ProDA for fair comparison.\n\n|Method|mAP|R@150|R@100|R@50|R@20|Params|Time(s)|\n|-|-|-|-|-|-|-|-|\n|STTran| 3.31| 71.29|62.91|42.67|22.14|-|-|\n|HORT|3.75|78.57|67.78|50.33|26.96|-|-|\n|SlowFast|5.00| 81.66|74.00|52.74|26.82|-|-|\n|ACARN|5.44|82.85|75.22|56.53|31.77|-|-|\n|**SportsHHI**(baseline)|10.69|89.25|82.93|68.13|43.72|111.18M|0.0478|\n|**SportsHHI\\***(reimplemented)|10.16 (↓0.53)| 90.07 (↑0.82) |84.83 (↑1.9)|68.40(↑0.37)|40.16(↓3.56)|123.11M|0.0484|\n|**Ours**(baseline+ProDA)|**13.55 (↑2.86)**|**92.54 (↑3.29)**|**87.91 (↑4.98)**|**72.69(↑4.56)**|**47.81(↑4.09)**|**123.19M**|**0.0520**|\n\n**Q2:** Could the authors include a qualitative visualization (e.g., t-SNE) comparing disentangled vs. unified features?\n\n**A2:** We appreciate the reviewer’s suggestion to include additional visualizations. However, due to the **rebuttal policy of NeurIPS 2025** which does not allow uploading additional figures, external links, or supplementary materials during the rebuttal stage, we are unable to provide the requested image at this time.\n\nWe would be happy to include the full visualizations and corresponding analysis in the final version of the paper if it is accepted.\n\n**Although we are unable to show the visualization, we computed the inter-class and intra-class distances after applying t-SNE for dimensionality reduction**. For each video, we randomly disentangle one action, meaning each video has a unified feature and a disentangled feature, covering a total of 157 action classes. For the disentangled features, the intra-class and inter-class distances are **0.624 ± 0.438** and **46.769 ± 24.966**, respectively. For the unified features, we only computed the intra-class distance, which is **28.841**. It can be observed that the intra-class distances for the disentangled features are mostly within 1, while the inter-class distances are much larger. This indicates that our disentangled features effectively enlarge the distances between different action categories, making them more separable in the feature space. Meanwhile, the smaller intra-class distances suggest that features within the same category are more compact and consistent, enhancing intra-class stability. In contrast, the larger intra-class distance of the unified features indicates a more mixed representation with less distinct class boundaries. Overall, this validates the effectiveness of our feature disentanglement in separating actions, providing more discriminative representations for subsequent action recognition and understanding tasks.\n\n\n**Q3:** Why is the evaluation limited to a single dataset?\n\n**A3:** We thank the reviewer for pointing this out. In the original submission, we evaluated our method on Charades to focus on Multi-label classification. We have additionally evaluated our method on the SportsHHI dataset , which contains more diverse and complex human-human interaction scenarios on Human-Human-Interaction Classification (HHICls). In our original manuscript, we conducted experiments on Charades (Action Genome), which primarily focuses on human-object interactions. To further validate the generalizability of our method to more complex human-human interaction scenarios, we have added additional experiments on SportsHHI , a challenging benchmark designed specifically for dense and diverse human-human interactions in sports videos. Compared to Charades (Action Genome), SportsHHI involves more frequent and overlapping interactions among multiple players, better aligning with our target task and demonstrating the robustness of our approach across different interaction types.\n\nAs results shown in **Table 1 above**. We observe that our method achieves a significant performance gain with only 10.80\\% increase in parameters. Notably, our DPM modules contribute only 1.46M parameters, accounting for just 1.18\\% of the total parameters. Under our reimplemented version with a parameter size comparable to our method, our approach still achieves a clear advantage (+3.39 mAP). This indicates that the performance improvement mainly stems from better action disentanglement rather than an increase in model capacity. \n\n**Q4:** How sensitive is ProDA to errors in SSG construction?\n\n**A4:** Thank you for the insightful question regarding the sensitivity of ProDA to noisy in the SSG construction. We follow the same setting as LaIAR , employing Faster R-CNN to detect and classify objects in videos, and to construct a noisy SSG (inaccurate object/relation predictions and missing temporal links for the same objects across frames ). As shown in Table 2, our method achieves the best performance even under the noisy SSG setting. Notably, the performance drop caused by the noisy SSG is smaller for our method compared to other approaches. We attribute this to our action disentanglement mechanism, which explicitly separates actions from the complex and error-prone scene context, allowing the model to reason over cleaner, disentangled representations. In contrast, feature extraction-based methods rely heavily on the error and entangled SSG, making them more sensitive to detection errors.\n\n**Table 2:** Results of SSGs under different conditions. GT SSGs are ground-truth SSGs without detection errors. FRCN SSGs are obtained using a detector (i.e., Faster R-CNN) and may contain errors.\n|Method|FRCNN SSGs mAP (%)|GT SSGs mAP (%)|\n|-|-|-|\n|LaIAR+NL|45.1|67.4 (+22.3)|\n|OR2G+NL|44.9|67.5 (+22.6)|\n|Ours (ProDA)|50.2| 71.1 (+20.9)|"}, {"review_id": "IcX7gJlD4K", "reviewer": "Reviewer_kb31", "summary": "The paper presents a framework for disentangling specified actions from multi-action video scenes using prompt-guided spatio-temporal scene graphs. By leveraging prompts to focus on particular actions, the method isolates relevant object and interaction representations, making it easier to model complex scene dynamics. This targeted disentanglement leads to improved performance on standard multi-label action recognition benchmarks.", "questions": "Already mentioned in weaknesses", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths : \n\n1. The paper is well written, and the motivation, key ideas, and contributions are clearly articulated and easy to follow.\n\n2. The idea of disentangling desired actions by leveraging prompts is very compelling. Based on the results provided, the approach appears to be effective. While vanilla methods may struggle in ambiguous scenarios, the proposed DPM framework seems to overcome these limitations by generating adaptive prompts conditioned on both the action specification and the video content.\n\n3. The ablation study is thorough and clearly isolates the contributions of each module, helping to attribute the performance gains to specific components of the framework.\n\nWeaknesses :\n\n1.  While the method shows some improvements in mAP, the added complexity of the proposed approach may not be fully warranted for the action recognition task, particularly in the absence of a detailed comparison of model efficiency or computational cost. In other words, can some of the performance gains be coming from increased model capacity rather than the disentaglement loss? \n\n2. Since the central claim of the paper is that explicit disentanglement of actions improves performance, a more compelling demonstration would involve applying the disentanglement loss to an existing baseline model without increasing the parameter count. This would better isolate the effect of the disentanglement itself. As it stands, the comparison in Table 3 involves different architectures and training setups, making it unclear whether the performance gains are due to the disentanglement mechanism or other architectural differences and whether the disentanglement loss is always strictly necessary for all models. \n\n3. The evaluation is limited to a single dataset, and the performance gains over certain baselines are relatively modest. Including results on one or two additional datasets would strengthen the generalizability claims and further validate the robustness of the approach.\n\n4. Although the core ideas are well-motivated, the transition between some concepts could be smoother. Improving the overall narrative flow would help the reader better follow the progression of the proposed framework.", "was_revised": true, "final_justification": "This paper presents an interesting new angle on the role of disentangled representations for the task of action recognition and I beleive the community would find this interesting.", "rebuttal": "We thank the reviewer for their feedback, and we are glad to see they have recognized the significance of our contribution. They have nonetheless asked some questions, which we address below.\n\n**W1:** Without detailed comparisons of model efficiency or computational cost, it remains unclear whether some of the performance gains arise from increased model capacity rather than the Action Disentanglement Loss?\n\n**A1:** We thank the reviewer for the insightful comment. To demonstrate that the performance gain of our method primarily comes from the proposed action disentanglement mechanism, rather than merely increased model capacity, we conduct additional experiments on SportsHHI , a recent benchmark focusing on human-human interaction in sports videos. It provides fine-grained annotations of interactions between players, SportsHHI contains 34 high-level interaction classes from basketball and volleyball sports. 118,075 human bounding boxes and 50,649 interaction instances are annotated on 11,398 keyframes. Compared to conventional action datasets, SportsHHI offers richer relational labels and poses greater challenges due to its dense, multi-person interactions.\n\nWe conduct two experiments for fair and thorough evaluation:\n- Module Integration: We extend the official SportsHHI baseline  (implemented with MMAction2 ) by integrating our proposed components, including two Dynamic Prompt Modules (DPM)—one for the Unspecified Action Prompt (UAP) and one for the Specified Action Prompt (SAP)—as well as the Action Disentanglement Loss (AD Loss).\n- Parameter-Controlled Baseline: To rule out the impact of model capacity, we construct a parameter-matched variant of the baseline by increasing its model size to match that of our approach, without incorporating any of our proposed modules.\n\nWe address the reviewer’s concern from the following two perspectives. \n\nFirst, as results shown in Table 1, we increased the parameter count by 10.8\\% (with the DPM contributing only 1.46M parameters, which is 1.18\\% of the total), yet achieved a significant accuracy improvement of +2.86 mAP. **Under this comparable parameter setting, our method significantly outperforms the baseline (+3.39mAP).** This highlights the generalization ability of our approach and further confirms that the performance gain mainly stems from the proposed AD Loss, rather than simply from increased model capacity. \n\nSecond, as shown in Table 2 (Evaluation on Charades Dataset), without adding the Action Disentanglement Loss (AD Loss), our method already achieves a competitive accuracy of 67.45 mAP, comparable to the previous state-of-the-art (SOTA) result of 67.5 mAP. Furthermore, in our ablation study under the same architecture and training settings, **incorporating the AD Loss improves the performance by +3.65 mAP (67.45 _v.s_. 71.10).** This demonstrates that the performance gain is largely attributed to the effectiveness of our disentanglement loss beyond the base model capability.\n\n**Table 1:**  Human-Human Interaction Classification results in SportsHHI. SportsHHI\\* is our reimplementation based on the official SportsHHI code, with minor modifications (e.g., parameter size) to match the capacity of ProDA for fair comparison.\n|Method|mAP|R@150|R@100|R@50|R@20|Params|Time(s)|\n|-|-|-|-|-|-|-|-|\n|STTran| 3.31| 71.29|62.91|42.67|22.14|-|-|\n|HORT|3.75|78.57|67.78|50.33|26.96|-|-|\n|SlowFast|5.00| 81.66|74.00|52.74|26.82|-|-|\n|ACARN|5.44|82.85|75.22|56.53|31.77|-|-|\n|**SportsHHI**(baseline)|10.69|89.25|82.93|68.13|43.72|111.18M|0.0478|\n|**SportsHHI\\***(reimplemented)|10.16 (↓0.53)| 90.07 (↑0.82) |84.83 (↑1.9)|68.40(↑0.37)|40.16(↓3.56)|123.11M|0.0484|\n|**Ours**(baseline+ProDA)|**13.55 (↑2.86)**|**92.54 (↑3.29)**|**87.91 (↑4.98)**|**72.69(↑4.56)**|**47.81(↑4.09)**|**123.19M**|**0.0520**|\n\n\n**Table 2:** Impact of AD Loss on the Performance of the Charades Dataset\n|Method|mAP (%)|\n|-|-|\n|OR2G+NL|67.5|\n|Ours w/o AD Loss|67.45|\n|Ours w/ AD Loss|71.10|\n\n**W2:** To better isolate the effect of disentanglement itself, it would be more appropriate to add the AD Loss to a baseline without increasing model parameters. As shown in Table 3 of the original manuscript, it is unclear whether the performance gains come from architectural changes or from the disentanglement mechanism. Furthermore, it remains uncertain whether the disentanglement loss is strictly necessary for all model variants.\n\n**A2:** We appreciate the reviewer’s valuable feedback. We apologize the Table 3 of the original manuscript did not clearly convey the intended comparison, and we clarify that settings ①-⑤ all use the same backbone and training configuration. We will explicitly clarify this in the final version of the paper, if it is accepted. **We conducted experiments on the SportsHHI dataset to verify the effectiveness of applying the disentanglement loss to an existing baseline model without increasing the number of parameters.**\n\nTo address the reviewer’s questions, we respond in two parts:\n- In the original ablation on the Charades dataset, adding the AD Loss improves mAP by +3.65 over the same model without it (As shown in **Table 2 above**). Additionally, we conducted new experiments on the SportsHHI dataset, where we added our proposed modules to the baseline. To control for parameter count, we also modified the baseline by expanding the FFN hidden dimension in the Transformer to match the parameter size of our method (As shown in **Table 1 above**). **The results show that our method still outperforms the parameter-matched baseline (+3.39mAP), suggesting that the performance gains stem not just from increased capacity, but from the effectiveness of our design.**\n- Regarding the AD Loss, it consists of two parts: a reconstruction loss and a disentanglement loss. Our goal is to explicitly disentangle action features, providing more concise and focused representations for specified actions, thus improving multi-label action understanding. **The reconstruction loss encourages the model to retain necessary information during separation. However, if only reconstruction is used, the model tends to preserve redundant features, undermining the purpose of action disentanglement.** Therefore, we introduce a disentanglement loss to minimize correlations between different action branches, ensuring that shared or redundant information is reduced.\n\n**W3:** The evaluation is limited to a single dataset, and the performance gains over certain baselines are relatively modest. Including results on one or two additional datasets would strengthen the generalizability claims and further validate the robustness of the approach.\n\n**A3:** We thank the reviewer for pointing this out. To address the reviewer’s concern and demonstrate the generalization ability of our approach,\n\nIn the original submission, we evaluated our method on Charades to focus on Multi-label classification.\nTo address the reviewer’s concern and demonstrate the generalization ability of our approach, we conduct additional experiments on another dataset, SportsHHI . In the original manuscript, we evaluated our method on Charades (Action Genome), which primarily focuses on human-object interactions. To assess the effectiveness of our approach in a different context, we choose SportsHHI, a dataset that emphasizes **human-human interactions**. Notably, SportsHHI is more complex and challenging, **as it centers on dense, multi-person interactions annotated in real-world sports videos.** In contrast, many conventional datasets (e.g., AVA ) contain fewer or relatively simpler human-human interactions.\n\nAs shown in **Table 1 above**, **Our method achieves a +2.86 mAP improvement with only a 10.80% increase in total parameters.** Notably, the introduced DPM modules contribute only 1.46M parameters—merely 1.18% of the overall model size—highlighting the lightweight nature of our additions. Furthermore, **compared to a parameter-matched baseline with a similar model size, our approach still demonstrates a clear advantage (+3.39 mAP)**. These results indicate that the performance improvement mainly stems from better action disentanglement rather than an increase in model capacity.\n\n**W4:** Although the core ideas are well-motivated, the transition between some concepts could be smoother. Improving the overall narrative flow would help the reader better follow the progression of the proposed framework.\n\n**A4:** We thank the reviewer for the constructive feedback and for recognizing the motivation behind our core ideas. We agree that improving the narrative flow and smoothing transitions between concepts can significantly enhance readability. We will carefully revise the manuscript to better organize the presentation and improve the transitions between key sections in the final version, if it is accepted.\nThese changes will help make the progression of our proposed framework clearer and more intuitive, facilitating a better understanding for readers.\n\nOur motivation is to disentangle specified actions from complex video scenes in order to better understand individual actions. To achieve this, we divide each video into two parts using a Specified Action Prompt (SAP) and an Unspecified Action Prompt (UAP), former is used to guide the model in disentangling the specified actions, while the latter disentangles all remaining actions. We use a Dynamic Prompt Module (DPM) to transform SAP and UAP into feature vector, which are then directly added to the video features to guide the model in disentanglement. To ensure that the model performs disentanglement rather than feature extraction, during training we enforce reconstruction of the original, non-disentangled features from the separated ones. Furthermore, to prevent excessive redundant information from hindering effective separation, we introduce a Disentanglement Loss that encourages more thorough disentanglement. For clarity, we include examples of how UAP and SAP are generated in the supplementary materials."}, {"review_id": "Tw0ILDD9WF", "reviewer": "Reviewer_Lz46", "summary": "This paper introduces ProDA, a framework that disentangles specified actions from multi-action videos using prompt signals and spatio-temporal scene graphs. The design is well structured, with components such as the Dynamic Prompt Module and a graph-based encoder that together achieve strong performance on standard benchmarks. The overall motivation is sound, and the system is practically useful for structured video understanding.\n\nHowever, the core idea of using prompts is only superficially explored. The prompts are implemented as simple multi-hot vectors without any semantic grounding, and the method does not incorporate language models that could reason about action semantics and generate richer prompt representations. This limits the conceptual depth of the work. While the prompt-guided framing is promising, its realization remains shallow. A more complete integration of language understanding could have made the contribution substantially stronger.\n\nOverall, this is a well-executed paper with a good idea at its core, but it lacks depth in exploring the full potential of prompt-based reasoning.", "questions": "1. The key concern is encoding prompts with fixed multi-hot vectors. This choice appears to block open-set recognition and prevents the framework from exploiting prior knowledge available in large language models, thereby limiting the broader significance of the contribution. Nevertheless, the work presents a well-executed and novel perspective on action recognition. If additional analysis is provided, such as evaluating the potential of language-model-guided prompts or demonstrating the flexibility of the current encoding scheme, I would be happy to raise my rating.\n\n2. Another concern relates to the fusion strategy used in Equation 1, where visual features and prompt features are combined through simple addition. This choice may limit the expressiveness of the interaction between action specifications and scene graph features. It would be valuable to explore alternative fusion methods such as concatenation followed by a linear layer, or cross-attention mechanisms. A comparison of these approaches could provide insight into whether the current additive design is optimal or if stronger interaction modeling would further enhance performance.\n\n3. VGNorm computes temporal statistics using mean-based aggregation to enforce consistency across frames. While this design is standard in normalization, it remains unclear whether mean alone is the most effective choice for modeling temporal dynamics in spatio-temporal scene graphs. Mean aggregation may dilute salient or bursty features that occur sparsely over time. It is not clear whether combining mean with other statistics such as max, or using a learnable aggregation function, would offer any advantage in this context. Could the authors clarify whether such alternatives were considered, and whether there is theoretical or empirical justification for relying solely on mean-based aggregation?\n\n4. The proposed method relies on object bounding boxes to construct spatio-temporal scene graphs, which serve as the foundation for downstream disentanglement. However, the quality of these bounding boxes during training—including their number, category diversity, and spatial distribution—is likely to have a significant impact on the effectiveness of feature disentanglement. The paper does not provide any analysis on how variations in bounding box quality influence model performance. Has this factor been studied, and is there any quantitative or qualitative assessment of how sensitive the method is to the quality of the input object annotations?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 2, "originality": 4, "strengths_and_weaknesses": "Strengths:\n1. The paper proposes a disentangled representation learning framework (ProDA) for multi-label action recognition, with the goal of isolating specific actions from complex scenes. The motivation is clear and addresses a practical limitation of prior methods that process all actions jointly.\n\n2. The framework is well structured, integrating modules such as the Dynamic Prompt Module (DPM), Video Graph Parsing Neural Network (VGPNN), and a novel normalization strategy (VGNorm) tailored to spatio-temporal scene graphs.\n\n3. The empirical results are strong, with consistent improvements over existing methods across multiple benchmarks. The ablation studies are thorough and support the utility of each component.\n\nWeaknesses:\n1. Although the paper frames the method as “prompt-guided,” the prompts are implemented using simple multi-hot vectors that lack semantic richness. This shallow encoding limits the model’s ability to reason about unseen or open-set actions, as it cannot generalize beyond the fixed label space.\n\n2. The method does not incorporate any language model or text encoder to provide semantic grounding for action prompts. As a result, the full potential of prompt-based guidance is underutilized, and the framework misses the opportunity to leverage external knowledge or compositional semantics.", "was_revised": true, "final_justification": "The authors have addressed all of my concerns, and the paper is solid overall. Therefore, I am raising my rating to accept.", "rebuttal": "We thank the reviewer for highlighting the strengths of our work. They have nonetheless asked some questions, which we address below.\n\n**Q1:** The key concern is encoding prompts with fixed multi-hot vectors. This choice appears to block open-set recognition and prevents the framework from exploiting prior knowledge available in large language models, thereby limiting the broader significance of the contribution. Nevertheless, the work presents a well-executed and novel perspective on action recognition. If additional analysis is provided, such as evaluating the potential of language-model-guided prompts or demonstrating the flexibility of the current encoding scheme, I would be happy to raise my rating.\n\n**A1:** We appreciate the reviewer for raising this point. In response, we conducted an additional experiment using textual prompts. We converted all Action Specification (AS) into text form and then used the CLIP (ViT-B/16)  text encoder (as used in our main method) to encode the textual HOI action name corresponding to each task. To adapt the textual prompts to our task, we appended a CLIP Adapter after the text prompt embeddings. Finally, we used an addition-based fusion, where the textual prompt was directly added to the visual features. Given an $AS  = \\\\{as_i\\\\}_{i=1}^K$, where each $as_i$ represents textual Human-Object Interaction (HOI) action name (e.g., \"Holding some clothes\", \"Putting clothes somewhere\"), the calculation process is as follows.\n\n\\begin{equation}\n\t\\text{text prompt}=\\mathtt{Adapter}(\\sum_{i=0}^{K}\\mathtt{CLIP}(as_i))\n\\end{equation}\n\nThe detailed results of this experiment are shown in Table 1. The results show that the multi-hot vector-based AS achieves a considerable advantage. This may be attributed to two factors:  1)the Action Specification (AS) includes many distractor items, making it difficult for textual-based AS to generalize effectively; 2) our multi-hot vector prompts generates T candidate prompts for each action category (we use $T=10$ in our method), which **provides greater representational capacity than a single prompt embedding. Moreover, we dynamically aggregate the candidate prompts based on both the input features and AS weights, enabling a more adaptive and informative representation**. Despite lacking the prior knowledge of large-scale language models, this approach still achieves strong performance. In future work, we plan to explore integrating large vision-language models into our framework to combine structural disentanglement with semantic generalization.\n\n**Table 1:** Comparison between textual encoding and multi-hot encoding.\n\n|Prompt Method |mAP (%)|\n|-|-|\n|Multi-hot Vector Prompts| 71.10|\n|Text Prompts|64.08|\n\n**Q2:** Another concern relates to the fusion strategy used in Equation 1, where visual features and prompt features are combined through simple addition. This choice may limit the expressiveness of the interaction between action specifications and scene graph features. It would be valuable to explore alternative fusion methods such as concatenation followed by a linear layer, or cross-attention mechanisms. A comparison of these approaches could provide insight into whether the current additive design is optimal or if stronger interaction modeling would further enhance performance.\n\n**A2:** Under the same training setups and model architecture, we additionally tested two alternative fusion methods (Results are shown in Table 2). The first method (line 2, Concate&Linear) concatenates the prompt and the original features, followed by a single linear layer with GELU activation to fuse them. The second method (line 3, Cross-Attention) employs a single-layer cross-attention module, where the prompt acts as the query (Q), and the original features serve as the key (K) and value (V). The results are shown in Table 2. We observe that both concatenation and addition yield similar performance, both outperforming the cross-attention approach. This may be due to the cross-attention introducing significantly more parameters, which increases training difficulty and the risk of overfitting. Comparatively, the addition method uses the fewest parameters and achieves the best results.\n\n**Table 2:** Results of different fusion methods in the DPM\n|Fusion Method|mAP (%)|\n|-|-|\n|addition| 71.10|\n|Concate & Linear|71.00|\n|Cross-Attention|70.04|\n\n**Q3:** VGNorm computes temporal statistics using mean-based aggregation to enforce consistency across frames. While this design is standard in normalization, it remains unclear whether mean alone is the most effective choice for modeling temporal dynamics in spatio-temporal scene graphs. Mean aggregation may dilute salient or bursty features that occur sparsely over time. It is not clear whether combining mean with other statistics such as max, or using a learnable aggregation function, would offer any advantage in this context. Could the authors clarify whether such alternatives were considered, and whether there is theoretical or empirical justification for relying solely on mean-based aggregation?\n\n**A3:** We additionally evaluated different aggregation strategies within the normalization component under the same training setups and model architecture. The first variant augments the standard mean aggregation with per-frame maximum values, i.e., computing the maximum within each frame and combining it with the mean. The second variant removes global averaging entirely and instead uses a learnable aggregation function, implemented as a one-layer MLP that predicts fusion weights to combine frame-level features.\n\nAs shown in Table 3 both Mean (line 1) and Mean&Max (line 2) aggregation yield similar performance, and outperform the MLP-based method (line 3). This may be due to the fact that the learnable MLP fails to maintain temporal consistency across frames. **While mean aggregation may dilute salient or bursty features that occur sparsely in time, we mitigate this issue by introducing learnable scaling parameters on the mean, which helps it match the performance of the mean+max approach.**\n\nConsidering both accuracy and computational efficiency, we adopt simple mean aggregation as the final choice in our framework.\n\n**Table 3:** Comparison of Aggregation Strategies in VGNorm\n|Methods|mAP (%)|\n|-|-|\n|Mean|71.10\n|Mean&Max|70.80|\n|MLP|67.18|\n\n**Q4:** The proposed method relies on object bounding boxes to construct spatio-temporal scene graphs, which serve as the foundation for downstream disentanglement. However, the quality of these bounding boxes during training—including their number, category diversity, and spatial distribution—is likely to have a significant impact on the effectiveness of feature disentanglement. The paper does not provide any analysis on how variations in bounding box quality influence model performance. Has this factor been studied, and is there any quantitative or qualitative assessment of how sensitive the method is to the quality of the input object annotations?\n\n**A4:** Thank you for the insightful question regarding the sensitivity of ProDA to errors in SSG construction. **We follow the same approach as LaIAR** by employing Faster R-CNN to detect and classify objects in videos, and construct SSGs with inherent errors. The detailed results are shown in Table 4. As can be seen, **our method achieves the best performance even when using noisy SSGs. Moreover, our performance drop is the smallest among all compared methods**. We attribute this to our action disentanglement mechanism, which explicitly separates actions from the complex scene context. This allows our model to understand actions in a simpler and less error-prone representation. In contrast, feature-based methods attempt to interpret actions directly from the noisier and more entangled SSGs.\n\n**Table 4:** Results of SSGs under different conditions. \nGT SSGs are ground-truth SSGs without detection errors. FRCN SSGs are obtained using a detector (i.e., Faster R-CNN) and may contain errors.\n|Method|FRCNN SSGs mAP (%)|GT SSGs mAP (%)|\n|-|-|-|\n|LaIAR+NL|45.1|67.4 (+22.3)|\n|OR2G+NL|44.9|67.5 (+22.6)|\n|Ours|50.2| 71.1 (+20.9)|"}, {"review_id": "NMJAnmslI1", "reviewer": "Reviewer_SVW3", "summary": "This paper proposes a Prompt-guided Disentangled Representation framework for action recognition. Experimental results on video action recognition demonstrate the effectiveness of the proposed method, ProDA, compared to state-of-the-art approaches.", "questions": "•The contributions and novelty of the proposed method should be better refined and reorganized to highlight its significance.\n\n• Existing multimodal large language models (MLLMs) have already demonstrated strong performance in action recognition, particularly in few-shot and zero-shot settings [1]. The authors should clearly state what advantages their method offers over such models.\n\n•In the proposed Action Disentanglement module, it is unclear whether the Action Specification captures temporal relationships. Would changing the order of specified actions or introducing interaction between actions affect model performance?\n\n•The idea of using prompts to augment video inputs and enhance model performance—i.e., prompt learning—has been widely explored in prior work. The authors fail to sufficiently discuss and cite relevant literature on this topic [3–5].\n\n•To validate the generalizability of the proposed method, additional widely used datasets such as AVA [2] should be considered.\n\n• The differences between VGNorm and previous GNN norm-based methods need to be clearly explained. If applicable, comparisons with more recent normalization techniques such as GRANOLA[6] should also be included.\n\nReference:\n1. LLaVAction: evaluating and training multi-modal large language models for action recognition\n2. Gu, Chunhui, et al. \"Ava: A video dataset of spatio-temporally localized atomic visual actions.\" Proceedings of the IEEE conference on computer vision and pattern recognition. 2018.\n3. Zhou, Kaiyang, et al. \"Learning to prompt for vision-language models.\" International Journal of Computer Vision 130.9 (2022): 2337-2348.\n4. Bahng, Hyojin, et al. \"Exploring visual prompts for adapting large-scale models.\" arXiv preprint arXiv:2203.17274 (2022).\n5. Khattak, Muhammad Uzair, et al. \"Maple: Multi-modal prompt learning.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2023.\n6. Eliasof, Moshe, et al. \"Granola: Adaptive normalization for graph neural networks.\" arXiv preprint arXiv:2404.13344 (2024).", "limitations": "The author did not discuss the limitations of the work.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "The method achieves the SOTA performance on multi-label action understanding with interpretability.\nHowever, the novelty of the proposed method should be better refined and reorganized, and more comparison with existing methods and evaluation on other datasets should be classified, as shown in the questions below.\nMoreover, there are some typo mistakes such as \"State-of-the Art Methods\" in line 298.", "was_revised": true, "final_justification": "The authors have added experiments and addressed most of my concerns, and I raise my score.", "rebuttal": "We thank the reviewer for the positive comments on our method. The reviewer also raised concerns, which we address below.\n\n**Q1:** The contributions and novelty of the proposed method should be better refined and reorganized to highlight its significance.\n\n**A1:** In the final version, we will revise the introduction and contributions section to more clearly emphasize the novelty of our method. Our key innovations include: 1) a prompt-guided disentanglement framework for multi-action videos; 2) a dynamic prompt module that adaptively encodes specified actions into expressive prompts to guide disentanglement; and 3) a tailored normalization and loss design for structured and temporally consistent action representation.\n\n**Q2:** Existing MLLMs already perform well in action recognition, especially in few-shot and zero-shot cases. The authors should clearly explain what advantages their method has over these models (e.g., LLaVAction).\n\n**A2:** While LLaVAction have shown promising performance in few-shot and zero-shot action recognition, our approach differs from LLaVAction  in several key aspects, which we summarize from the following two perspectives:\n- **Motivation.** Our goal is to improve video understanding by disentangling actions from complex scenes. We focus on modeling human-object and human-human interactions, enabling more interpretable reasoning over interaction structures. In contrast, LLaVAction addresses the limited generalization of egocentric video datasets by converting class labels to natural language. They find that adding distractors during test harms performance on multi-choice QA tasks and propose method to mitigate this.\n- **Task Scope.** Our approach constructs SSGs and uses subgraph parsing to disentangle actions, yielding a framework that is naturally interpretable and extensible. For example, localization arises directly from attention weights during graph readout, without explicit supervision. In contrast, LLaVAction requires supervision for localization.\n\nDue to limited resources and time during the rebuttal period, we were unable to conduct a full evaluation using LLaVAction . **Instead, we carried out an experiment by replacing our multi-hot vector prompt with a CLIP-based text prompts.** Specifically, we converted all class labels into natural language and used CLIP to encode them as prompts. As shown in Table 1, our method achieves a significant performance advantage (+7.02 mAP). We attribute this to our method's ability to generate multiple prompts per action specification (AS) and dynamically aggregate them, resulting in stronger representational capacity. \n\n**Table 1:** Comparison between Multi-hot Vector Prompts and Text Prompts.\n|Prompt Method |mAP (%)|\n|-|-|\n|Multi-hot Vector Prompts| 71.10|\n|Text Prompts|64.08|\n\n**Q3:** In the proposed Action Disentanglement module, it is unclear whether the Action Specification captures temporal relationships. Would changing the order of specified actions or introducing interaction between actions affect model performance?\n\n**A3:**  Through **our frame-level localization results**, we observe that the model is able to **capture the temporal extent of actions**, indicating that the prompts effectively encode relationships among relevant atomic motions.\nSince the disentanglement prompts are represented as multi-hot vectors, changing the order of actions within the prompt does not affect the results.\n\nTo verify that changing the order of specified actions or introducing interactions between actions does not affect model performance, we randomly shuffled the label meanings in the dataset. For example, the label \"c000'\" originally represented \"Holding some clothes\", after shuffle, \"c000\" represents \"Sitting in a chair\". We tested results under multiple random seed shuffles, the results are shown in Table 2. **We observe no significant difference in performance, indicating that our model is invariant to the order of specified actions, and that potential interactions between action specifications do not significantly impact the model's effectiveness.**\n\n**Table 2:** Comparison of model performance using original _v.s._ shuffled dataset label.\n|Label Type|mAP (%)|\n|-|-|\n|Original|71.10|\n|Shuffled|71.50|\n\n**Q4:** The idea of using prompts to augment video inputs and enhance model performance—i.e., prompt learning—has been widely explored in prior work. The authors fail to sufficiently discuss and cite relevant literature on this topic [3–5].\n\n**A4:** While prompt learning is widely studied, prior work mainly adapts pretrained models for zero- or few-shot tasks using language prompts. In contrast, we use structured multi-hot prompts to explicitly guide action disentanglement in multi-label recognition. Our model is trained from scratch and does not rely on any pretrained language or vision models. We will clarify this distinction in the related work section.\n\n**We conducted additional experiments using a CLIP-based text encoder**. Specifically, we converted all action labels into natural language form and encoded them with CLIP to replace the original multi-hot vector prompt representation.  As shown in **Table 1 above**, our proposed method significantly outperforms the CLIP-based variant.\n\n**Q5:** To validate the generalizability of the proposed method, additional widely used datasets such as AVA  should be considered.\n\n**A5:** We conduct our experiments on SportsHHI instead of AVA. While AVA mainly contains low-density interactions in movie scenes, **SportsHHI captures dense, real-world Human-Human Interactions (HHI) from basketball and volleyball games, making it more challenging.** Thus, SportsHHI serves as a better benchmark for evaluating our method’s effectiveness in complex interaction scenarios.\n\nBased on the baseline, we incorporated our Dynamic Prompt Module (DPM) and Action Disentanglement Loss (AD Loss). To ensure a fair comparison, we conducted an additional experiment by increasing the parameter size of the baseline model to match that of our modified version. The detailed results are shown in Table 3. **Our method consistently outperforms the baseline across all metrics, demonstrating the generalization ability of our approach. Even under comparable parameter settings, our method still achieves significant performance improvements.** This indicates that the performance gain primarily stems from our disentanglement mechanism, rather than merely an increase in model capacity.\n\n**Table 3:** Human-Human Interaction Classification results in SportsHHI. \nSportsHHI\\* is our reimplementation based on the official SportsHHI code, with minor modifications (e.g., parameter size) to match the capacity of ProDA for fair comparison.\n|Method|mAP|R@150|R@100|R@50|R@20|Params|Time(s)|\n|-|-|-|-|-|-|-|-|\n|STTran| 3.31| 71.29|62.91|42.67|22.14|-|-|\n|HORT|3.75|78.57|67.78|50.33|26.96|-|-|\n|SlowFast|5.00| 81.66|74.00|52.74|26.82|-|-|\n|ACARN|5.44|82.85|75.22|56.53|31.77|-|-|\n|**SportsHHI**(baseline)|10.69|89.25|82.93|68.13|43.72|111.18M|0.0478|\n|**SportsHHI\\***(reimplemented)|10.16 (↓0.53)| 90.07 (↑0.82) |84.83 (↑1.9)|68.40(↑0.37)|40.16(↓3.56)|123.11M|0.0484|\n|**Ours**(baseline+ProDA)|**13.55 (↑2.86)**|**92.54 (↑3.29)**|**87.91 (↑4.98)**|**72.69(↑4.56)**|**47.81(↑4.09)**|**123.19M**|**0.0520**|\n\n**Q6:** The differences between VGNorm and previous GNN norm-based methods need to be clearly explained. If applicable, comparisons with more recent normalization techniques such as GRANOLA should also be included.\n\n**A6:** We compared widely used normalization techniques in graph learning, including BatchNorm, LayerNorm, and GraphNorm, specifically in the context of SSG (As shown in Table 4).\n- During training, BatchNorm computes statistics over the mini-batch, which can suppress sparse yet informative frame-level features in temporal data. To address this, we introduce learnable scaling for the mean and compute variance in real-time on each input to better preserve salient temporal signals.\n- LayerNorm operates at the node level, normalizing each node independently, whereas our approach performs normalization globally over the entire SSG layer.\n- GraphNorm computes the mean in real-time and applies learnable parameters to preserve salient features per graph. However, it lacks normalization across the temporal batch dimension. To address this, we use the accumulated mean during training, enabling temporal batch-level normalization.\n\n**Table 4:** Differences among LayerNorm, BatchNorm, GraphNorm, and VGNorm.\n\"Computation of Mean\" and \"Computation of Std\" refer to how the mean and standard deviation are computed. \"In-Batch\" indicates that the statistics are calculated from the current input batch, while \"In-Training\" uses accumulated statistics during training. * indicates that a learnable scaling parameter mean_scale is applied to the mean during normalization.\n||LayerNorm|BatchNorm|GraphNorm|VGNorm|\n|-|-|-|-|-|\n|Norm Aspect|Node|Batch|Graph|Batch & Graph|\n|Computation of Mean|In-Batch|In-Training|In-Batch*|In-Training*|\n|Computation of Std|In-Batch|In-Training|In-Batch|In-Batch|\n\nAs the official code for GRANOLA is currently unavailable (the GitHub repository linked in the paper is empty), we implemented our own version for evaluation. The results are shown in Table 5.**Our method achieves better results**. Compared to GraphNorm, GRANOLA also performs normalization at the graph level. GRANOLA leverages Random Node Features (RNF) to adaptively capture the structural properties of input graphs, and generates dynamic affine transformation parameters through GNN and MLP modules, thereby enhancing its expressive power. However, **GRANOLA ignore normalization across the temporal dimension**, which limits their effectiveness in video-based settings.\n\n**Table 5:** Comparison between textual encoding and multi-hot encoding.\n|Norm Type|mAP (%)|\n|-|-|\n|VGNorm|71.10|\n|GRANOLA| 70.29|\n|GraphNorm|68.61|"}]

Author rebuttal:

Meta-review: The paper proposes an approach that converts a short video clip into a scene graph. Given a pre-defined set of actions, it then identifies the sub-graph of the scene that corresponds to these actions. The reviewers appreciate that the problem, which is addressed in the paper, is well motivated and addresses a practical limitation of prior methods. Reviewer kb31 considers the idea of disentangling desired actions as compelling and acknowledges the thorough ablation studies. The reviewers, however, had several concerns. A major concern has been the experimental evaluation. It is performed on a single dataset and the impact of the quality of the spatio-temporal scene graph is not evaluated. There has been also concerns regarding the increased complexity compared to standard multi-label action recognition models and the unclear impact of the disentanglement loss, including a visualization that shows the disentanglement of the representation compared to conventional features. Finally, Reviewer SVW3 points out that the method does not use prompts but hot-vector encoded action  classes. The rebuttal provided additional results for the SportsHHI dataset and an analysis of the impact of the quality of the spatio-temporal scene graph. Preliminary results using text prompts have been provided as well, but these are not mature enough. After the rebuttal, the reviewers unanimously recommend the acceptance of the paper.

The AC acknowledges as the reviewers the additional results that have been reported in the rebuttal. Given the motivation of the problem and the shown benefit of the disentanglement loss, the AC agrees with the reviewers that the paper can be accepted. The manuscript, however, needs to be substantially improved by adding the new results, demonstrating the disentanglement (e.g. by adding a visualization), reconsidering the term 'prompt', and improving the reading flow as pointed out by Reviewer kb31.
Decision: Accept (poster)