HopaDIFF: Holistic-Partial Aware Fourier Conditioned Diffusion for Referring Human Action Segmentation in Multi-Person Scenarios
Authors: Kunyu Peng
Junchao Huang
Xiangsheng Huang
Di Wen
Junwei Zheng
Yufan Chen
Kailun Yang
Jiamin Wu
Chongqing Hao
Rainer Stiefelhagen
Venue: NeurIPS 2025 spotlight ()
Keywords: Human action segmentation
video understanding
Primary area: applications

Abstract: Action segmentation is a core challenge in high-level video understanding, aiming to partition untrimmed videos into segments and assign each a label from a predefined action set. Existing methods primarily address single-person activities with fixed action sequences, overlooking multi-person scenarios. In this work, we pioneer textual reference-guided human action segmentation in multi-person settings, where a textual description specifies the target person for segmentation. We introduce the first dataset for Referring Human Action Segmentation, i.e., RHAS133, built from 133 movies and annotated with 137 fine-grained actions with 33h video data, together with textual descriptions for this new task. Benchmarking existing action segmentation methods on RHAS133 using VLM-based feature extractors reveals limited performance and poor aggregation of visual cues for the target person. To address this, we propose a holistic-partial aware Fourier-conditioned diffusion framework, i.e., HopaDIFF, leveraging a novel cross-input gate attentional xLSTM to enhance holistic-partial long-range reasoning and a novel Fourier condition to introduce more fine-grained control to improve the action segmentation generation. HopaDIFF achieves state-of-the-art results on RHAS133 in diverse evaluation settings. The dataset and code are available at https://github.com/KPeng9510/HopaDIFF.

HopaDIFF: Holistic-Partial Aware Fourier Conditioned Diffusion for Referring Human Action Segmentation in Multi-Person Scenarios
Kunyu Peng 1.2
Junchao Huang?
Xiangsheng Huang* Di Wen' Junwei Zheng' Chongqing Hao" Rainer Stiefelhagen'
Yufan Chen
Kailun Yang²
Jiamin Wu
Karlsruhe Institute of Technology
PINSAI,, Sofia University "St. Kliment Ohridski" *Institute of Automation, Chinese Academy of Sciences Shanghaii AI Lab HEEBUST
Beijing Institute of Technology
Hunan University
Abstract
Action segmentation is challenge in high-level video understanding, aim- ing to partition untrimmed videos into segments and assign each a label from a predefined action Existing methods primarily address single-person activities with fixed action sequences, overlooking multi-person scenarios. In this work, we pioneer textual snterence-guiderr human action segmentation in multi-person settings, where a textual description specifies the target person for segmentation. We introduce the first dataset for Referring Human Action Segmentation, i.e., RHAS133, built from 133 movies and annotated with 137 fine-grained actions with 33h video data, together with textual descriptions for this new task. Bench- marking existing action segmentation methods on RHAS133 using VLM-based feature extractors reveals limited performance and poor aggregation of visual cues for the target person. To address this, we propose holistic-partial aware Fourier- conditioned diffusion framework, i.e., HopaDIFF, leveraging a novel cross-input gate attentional xLSTM to enhance holistic-partial long-range reasoning and a novel Fourier condition to introduce more fine- ceee-grained control to improve the action segmentation generation. HopaDIFF achieves state-of-the-ard results on RHASI33 in diverse evaluation settings, The dataset and code are available at https:/ / /t///gttuu////// com KP Kengg551 10s51100HHHoDD.....
Introduction
Human action recognition [6, 42, 38, 56, 45, 20] is core problem with applications in robotics [44], surveillance [53], and healthcare [43]. While recent methods have shown strong performance, they typically rely on pre-trimmed video clips, which demand extensive manual pre-processing and limit applicability in real-time or continuous scenarios. Human action segmentation [14, 31, 57, 60], by contrast, operates untrimmed video streams and aims identify both action classes and their temporal boundaries. This setting is more challenging but aligns more closely with real-world use cases, where continuous and online interpretation of human behavior is essential.
However, existing human action segmentation approaches [40, 34. 2, 18] are predominantly designed for single-persen scenarios, where the performed actions follow predefined sequence dictated by specific task protocol, e.g., assembling tool [47] based on fixed instructions or preparing salad [50] according to set menu. Such settings lack the complexity and variability present in real-world daily life scenarios, where multiple individuals may appear in the same untrimmed video, and the action sequences are not pre-defined but exhibit significant randomness and spontaneity. This limitation
"Correspondence: xiangsheng.huang www.gghhaannnna.c..... @ia.ac.cn
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Textural Reference for Person I:
The boy with short golden hair.
Sitting Watching
N/A
Sitting MoveHead
N/A
Movellead
N/A
Sitting Watching
Walking
1,567
1,622
1,784
1,908
2,522
2,559 2,594
2,987
3,463
Textural Reference for Person 2: The old mon with wiile hair in brown shirt.
Sitting Looking
N/A
Sitting
N/A
Frowning
N/A
Speaking
ShakingSth Speaking
Walkking
ShakingSSth
Spesking
Speaking ShakingSth
Bending MoveEye
Speaking
1,567
1,724
1,788
2,075
2,146
2,246
2,333
2,395
2,448
2,521
2,560
2,593
2,987
3,172
Figure 1: An illustration of the referring human action segmentation task is shown. The example is taken from the RHAS133 dataset, with annotations provided for two referred individuals. The numbers beneath each annotation indicate the frame indices marking the start and end of each action. N/A denotes that the referred person is not present within the corresponding frame interval.
hinders the generalizability of current human action segmentation methods to more dynamic and unconstrained environments.
In this work, we introduce novel task, termed Referring Human Action Segmentation (RHAS), which, to the best of our knowledge, is the first to address action segmentation in multi-person untrimmed videos guided by textual references. We contribute the RHAS133 dataset, which comprises 133 untrimmed videos featuring 542 individuals, where each person of interest is annotated with textual reference. This work is inspired by prior works on referring video and image understanding [42]. These textual descriptions serve as guidance for the model to localize and segment the actions of the referred individuals, with fine-ge greigraiddd, multi-label action annotations provided the frame level following the AVA dataset format [21], as shown in Fig. 1.
We further adapt several well-established human action segmentation methods, i.e., FACT [40], ActD- iff [34], ASQuery [18]. and LTContent [2], as well as one referring human action recogmition method, i.e., RefAtomNet [42]. to the proposed RHAS setting and evaluate their performance on our newly constructed dataset, which serves as the first RHAS benchmark. However, the experimental results reveal that these baseline methods achieve limited performance under the RHAS framework. This can be attributed to the fact that existing human action segmentation approaches are primarily designed for single-person scenarios without textual guidance and are not equipped to handle unstructured action sequences without fixed protocol in complex, multi-person movie environments. Meanwhile, RefAtomNet [45], originally developed for referring action recognition trimmed videos, lacks the capacity for temporal modeling and long video understanding required by the RHAS task.
To improve referring human action segmentation, we introduce HopaDIFF, novel framework that employs a holistic- and partial-aware Fourier conditioned diffusion strategy guided by untrimmed video and textual references. consists of two branches: holistic branch that captures global context from the full vidco, and a partial branch that leverages GroundingDINO [37] to detect the referred person, generating a cropped video stream for fine-graine ane-grained representation using Aision-Language Model (VLM), e.g., BLIP-2 [29]. To facilitate effective long video reasoning and holistic- partial aware information exchange, we introduce a novel module, i.e., HP-XLSTM. which fuses holistic and partial features using cross-input gate attentional SL..M. Furthermore, to exert finer control over the generative process of the diffusion model for action segmentation, we incorporate an additional conditioning mechanism from the frequency domain based on the embeddings delivered by the HP-xLSTM. This is achieved by applying a Discrete Fourier Transform (DFT) along the temporal axis to extract trequency-basee cues, which are combined with original feature conditions to guide the denoising process and are used to preserve temporal coherence and fine-grained details on motion dynamics by guiding the model maintain consistency across both low and high-frequency components. The final segmentation prediction is obtained by averaging the denoised outputs from both branches, ensuring robust and contextually informed result. Extensive experiments demonstrate that HopaDIFF achieves ofate-of-the-as -hethrrtt performance on the proposed RHAS benchmark, significantly outperforming existing baselines in diverse evaluation settings.
To summarize, this paper makes three key contributions: (1) we introduce, for the first time, the task of Referring Human Action Segmentation (RHAS), paving the way for action segmentation in

---

complex, multi pertsneeoonn scenarios guided by natural language reference; (2) we present RHAS133, the first dataset for this task, and establish a comprehensive benchmark using diverse evaluation settings and adapted baseline methods; (3) we propose HopaDIFF, diffusion-based framework tailored for RHAS, featuring the novel HP-xL.STM module for long-term temporal modeling through cross-input gate attention between holistic and partial branches together with additional Fourier condition. Our method achieves state-of-the-a performance on the RHAS133 dataset.
Related Work
Referring Scene Understanding, Referring scene understanding, which leverages natural language to identify specific regions or objects within images or videos, has emerged as a valuable paradigm across various computer vision applications such as autonomous driving [58] and fine-grained action understanding [42]. The progress in this area has been greatly driven by the availability
of high-quality, open-source datasets and benchmarks [5, 61, 52, 26, 48, 58, 33], which provide standardized evaluation protocols and diverse referring scenarios. For instance, Li et al. [30] addressed referring image segmentation using a recurrent refinement network, whereas the CLEVR R-RRf++ dataset introduced by Liu et al. [36] focused on visual reasoning with structured referring expressions, More recently, Wu al. [58] proposed a benchmark for referring multi-object tracking, further broadening the scope of this research direction. Referring video segmentation [35, 59, 24, 65, 11, 17] is related research direction, primarily targeting pixel- wise semantic segmentation of a specified object within trimmed videos. In contrast, our task focuses per-frame action recognition to determine the temporal boundaries of different actions. Peng et al. [42] proposed the first benchmark to facilitate referring fine-grained human action recognition, yet their task is restricted to trimmed videos. In this work, we for the first time, conduct research on referring human action segmentation, which aims achieving concrete action understanding untrimmed videos. Since existing human action segmentation datasets [3, 27, 50] focused on single-person person scenarios and consist of action sequences recorded according to restricted protocol, which suitable for the exploration of referring human action segmentation, we collect the first RHAS dataset, i.e., RHAS133, from 133 different movies in multi-person scenarios with textual references. Our contributed dataset will serve important research foundation for the RHAS task.
Video-based Human Action Segmentation. Human action segmentation is challenging video understanding task that involves dividing long, untrimmed sequences into frame-level action seg- ments [10, 40, 34, 16, 63]. Unlike action recognition trimmed clips with single labels, segmentation requires precise frame-wiss wise predictions to model temporal boundaries. Common approaches lever- age temporal convolutional networks [28, 22, 31, 19, 32, 41, 64], graph neural networks [25, 62], and transformers [60, 55, 12, 13, 51, 39] to model long-range dependencies. Multi-stage rea- soning [60, 1, 31, 54, 14, 41, 57) further refines predictions iteratively. Recent methods such as FACT[400, sparse transformers(2], ASQuery [18), and diffusion models [34] have advanced the field. However, they mostly designed for single-person settings with fixed action protocols, overlooking multi-person complexities. In this work, we tackle referring human action segmentation by adapting recognition models with vision-language features (e.g.. BLIP-2 [29], CLIP [46]) for textual guidance. We propose HIFaDIFF, a diffusion-based model that fuses holistic and partial cues via cross-input gated attentional xLSTM and incorporates frequency-done domain conditioning. Our method sets new state of the art on the proposed RHAS133 dataset.
3 Benchmark
Dataset. We present RHAS133, the first dataset specifically curated for the task of referring human action segmentation. It comprises 133 movies featuring diverse, multi-person scenarios. Annotations follow the AVA protocol [21], with the action label set extended from 80 to 137 fine-grained classes to capture broader range of human behaviors. Textual references and action annotations are manually labeled and cross-validated by 6 annotators with domain expertise to ensure high annotation quality. The dataset spans 33 hours of video and includes 542 annotated individuals. Notably, the textual references describe target individuals without disclosing their actions.
Table compares RHAS133 with existing human action segmentation datasets. Prior datasets, such as SALADS50 [50], Assembly 101 [47], EgoProcel. [3], Breakfast [27], and GTEA [15], are typically limited to single-pent mnglee-person settings, lack natural language references, and rely on predefined
3

---

Table A statistical comparison with existing human action segmentation datasets, where TextRef. denotes the availability of textual reference annotations, FreeProo. indicates whether the dataset collected under a free-form procedure rather than a fixed action sequence, and NumClasses and NumPersons represent the total number of action classes and the number of participants.
Datasets
MultiPerson
TextRef.
MuttiLabel
FreeProc.
View
Duration
NumClasses
NumPersons
SALADS50 [50]
x
x
Ego-centric
50
Assembly 101 [47]
X
x
x
x
Ego-centric
513h
101
53
EgoProceL [3]
X
x
x
Ego-centric
62h
16
130
Breakfast (27)
X
x
X
x
Third-person
77h
10
52
GTEA [15]
X
x
x
x
Ego centric
20th
20
Ours
V
Third-person
33h
137
542
bridge whi
wear g Lasses
g
te beard
bluue
Phrckerdd a
acket
eyes the
Long
...ee
g
Light
ye
boy
head bang
shor
robe
mose
e
eeching
man
aged
cur
welanes
Ty
evebr
shi
cl
dark
foadd
clotheses
es
Idress
womans
S
Suit
black
shout
der
middle
gray
maai
br owww
Face secate
high
ponytall
weared
An illustration of the statistics of the dataset. The figure the left-hand side shows the number frames per action category, and the figure on the right the word cloud generated based on the textual annotation in RHAS133 dataset. Zoom in for better view.
Figure 2:
procedural action sequences. The per-action class statistics and word cloud of the textual reference annotation are shown in Fig. 2. contrast, RHAS133 uniquely combines multi-person interactions, textual references, and fine-grained action annotations in unstructured, third-person video contexts. These properties make it more comprehensive and challenging benchmark for advancing research in referring human action segmentation. Additional statistics are provided in the supplementary materials.
Baselines. To construct the first RHAS benchmark, we select baselines from both the human action segmentation and referring human action recognition domains comprehensively evaluate perfor- mance under this novel setting. From the human action segmentation field, include FACT [40], ActDiff [34], LTContent [2], and ASQuery [18], as they represent state-of-the-art methods employing diverse strategies such dual-branch cross-attention (FACT), diffusion-based action segmentation model (ActDiff), sparse long-range transformer (LTContent). and query-based segmentation with boundary refinement (ASQuery). These models chosen for their demonstrated ability to han- dle long, untrimmed videos and capture temporal dependencies, which are the key requirements for RHAS. In addition, include RefAtomNet [42] from the referring human action recognition domain due to its effective use of vision-language models for textual grounding, which aligns well with the core challenge in RHAS. These selected baselines allow limitations of both traditional segmentation techniques and reference- guided approaches under the RHAS framework.
the capabilities and
Methodology
4.1 Preliminaries.
Our proposed HopaDIFF is diffusion- based action segmentation model, as shown in Fig. 3. Diffusion models [23, 49] aim to approximate the true data distribution q through a parameterized model pa. These models comprise two key components: the forward process, which progressively corrupts input data with Gaussian noise over time steps following predefined variance schedule r = (~+ / E [], and the reverse process, which employs neural network to iteratively reconstruct the data from noisy observations. During the forward process, corrupted sample Vāzzo + EVI āte where a = Y and denotes the
at time step is computed

---

Vidro
VI.M Backhone
Holssiic Branch
HPSXLSTM
condition 0оо000
P Bauna
Ground Truth
mima
Decader
Prediction
f Modeling
condition
Fiee.
Holistic
Features
Backbone
condition I
Ground Truth
The wearing blue dress.
Textual Ref.
Dinn
Grounding
Crop& Resize
Encoderr
Modeling
Decodar
Prediction
i f
condition
Freg.
Patial
Features
Figure 3: An overview of the proposed HopaDIFF, which integrates two complementary diffusion- based branches, i.e., holistic and partial branches for action segmentation with target-referenced awareness. To improve controllability and segmentation precision, we introduce HP-xLSTM, cross-input gated module designed for effective exchange between holistic and partial features, and propose novel Fourier-based conditioning mechanism inject frequency-domain control signals into the generative process. During training, the two branches are individually supervised using ground-trune action labels and temporal boundary annotations.
cumulative product of a time. This formulation allows 2, to be directly sampled from the original data zo in closed form, enabling a reparameterized expression of z suitable for optimization. The reverse process seeks reverse this diffusion by gradually denoising the noisy sample at 7 step back to clean sample at / = 0 step. Each step of the reverse trajectory can be formulated as:
po (o(2-11/2) (i o
where the mean μ is predicted by a neural network, and of is determined by the variance schedule.
To model the denoising process, neural network Ma is trained to estimate the original clean signal from noise-corrupted counterpart. This network serves as the core of the reverse process by parameterizing the reconstruction procedure as defined in Eq. 2.
2 Vā, 1 V ā,-1 of
Vo.Mo((z. o,c.
(2)
VI az
Here, the timestep t is sampled uniformly during training, and the loss encourages accurate recon- struction of zo from z,. During inference, sample Zo is generated by starting from Gaussian noise Zr ~ (0, I) and iteratively applying the denoising step. Iterating Eq. 2 from t = T to = 0 yields sample Zg from Po (zo). This formulation serves the foundation for the existing diffusion-basse action segmentation approach [34] and our HopaDIFF, enabling flexible conditioning and iterative refinement in the temporal domain.
4.2 HopaDIFF
Diffusion- based Action Segmentation. ActDiff [34] first adopted diffusion to achieve the action segmentation, which aims to learn how to denoise the action segmentation ground truth iteratively perturbed by Gaussian noise during training time and generate concrete action segmentation labels based on random noise and temporal feature conditions during inference. To capture the underlying distribution of human actions, the model is trained to reconstruct the original action sequence from its corrupted form. During each training iteration, a random diffusion step t 21.,...., is selected. Subsequently, noise is added to the original ground truth sequence Yn based on cumulative noise schedule, resulting in corrupted version y, E [0, LXCC which is defined as y: Vāuyo + cv1 ā; and c denotes Gaussian noise. The denoising decoder Iv is trained to handle sequences with varying noise levels, including fully random noise. In inference, it begins with pure noise sample and progressively removes noise. The update at each step follows Eq. 2. This iterative process produces a sequence VT Yo). with yn as the final prediction approximating the ground truth. Though ActDiff shows promising performance the RHAS task, the lack of target-aware partial feature reasoning and the low granularity of control for the action segmentation generation limit its performance on the RHAS task.
5

---

To facilitate effective long-term information modeling and the exchange between holistic and par- tial cues related to the referred target, and to enhance the controllability of action segmentation generation conditioning, we propose a novel diffusion-based framework for referring human action segmentation, termed HopaDIFF. HopaDIFF a novel diffusion- framework for RHAS that integrates both holistic and partial video cues, guided by untrimmed video and corresponding textual references. It employs a dual-branch architecture, where the holistic branch captures global temporal context, and the partial branch focuses on fine-grat fine-grained representations of the referred person using DIOO [37] and VLM features from BLIP-2 [29]. To enhance temporal reasoning and controllability, HopaDIFF introduces a pross-input gate attentional module, i.e., HP-xLSTM, for holistic-spatii spatial feature exchange, and incorporates a reqquency-oomall conditioning mechanism using DFT to guide the diffusion-based action segmentation process.
Holistic and Partial Conditioned Diffusion for RHAS. HopaDIFF consists of two branches, ie., holistic and partial reasoning branches. For the holistic reasoning branch, the original video, i.e. Va. is fed into a VL.M-based cross-modal feature extractor (denoted as F ()) together with textual reference, i.c., to achieve the preliminary cross modal feature extraction. For the partial reasoning branch, the video and textual reference are first fed into GroundingDino (G-Dino) [37] to achieve the detection of the person of interest, and then the cropped frames based on the bounding boxes prediction are grouped and resized to formulate a new video, fed into the same VLM feature extractor later. Two encoders (E,, Ep) with the same structure are used for these two branches, where the parameters are not shared. Then the cross cross-modal modal feature conditions can be achieved by Eq. 3.
Zh, 2p = E,, (F (VV, r)), (F (G-Dino(V,; r))
where 2n and 2p denote holistic cross-modal features and partial aware cross-modal conditions.
To achieve better long-range modeling and temporal feature aggregation, we propose holistic-partial aware XL.TTM temporal reasoning, which can achieve cue exchange between both the holistic branch and the partial branch through bidirectional pross-inpul gate attention, abbreviated as HP-xLSTM. The enhanced features from those two branches can be calculated according to Eq. 4
2 2 HP-xLSTM((? 22" 2²))
Next, we will introduce the details regarding our proposed HP-xLSTM.
Holistic-Partial Aware xLSTM Temporal Reasoning. Extended LSTM (XLSTM) [4] introduces two major components, i.e., exponential gating and enhanced memory structures. These give rise to two variants, i.e., sLSTM, which uses scalar memory with mixing, and mLSTM, which employs matrix memory with parallelizable outer outer-product update. While mLSTM drops recurrent memory mixing which can be used to enable parallelism, sLSTM supports multiple cells and heads with selective memory mixing across cells. The holistic partial feature exchange is conducted at the input gate of the mLSTM block, as Eq. 5.
C.t. = =1 ffmCCpm C + 17" D 1TTT
mm& E {h. p}
cell states
n2?Th f(T n2th + +17h km
normalizer state
h
t ( h:
h: = C q:th) max {|nm" g.ll,
hidden state
4 m
Wm mny" t
+ bm
query input
k.
Wm z
m
key input
Vd
o W
21
value input
i = exp
P
BCA(
w
AT
z/
+
b
w
21 + DP)
input gate
fm = a fm
OR exp
mm
f.m
T
m 21
+ bf
forget gate
of = a (ō
W
92
m
output gate,
where BCA indicates the bidirectional cross- attentieeeonnnna conducted on the input gates of the holistic and partial feature reasoning branches.
Discrete Fourier Transformation based Frequency Conditions. Since action segmentation fun- damentally relies on per-frame action recognition to produce temporally coherent segmentation

---

outputs, a high degree of control granularity is essential when employing diffusion models to generate segmentation results from Gaussian noise, particularly when using features as conditions. Existing approaches, e.g., ActDiff [34], primarily leverage features in the temporal and spatial domains as conditioning inputs, while overlooking the potential advantages offered by the frequency domain. In this work, we explicitly incorporate Siqquencc-domail information, derived from features aggregated via -LSMM, to enhance the fine-grained controllability of the diffusion process and the capability of the model to deal with complex periodic temporal patterns. The frequency-basee conditions for both the holistic and partial branches are computed according to Eq. 6.
a 2 DFT(2"), DFT(2"),
where DFT indicates the discrete Fourier transform. We adopt two decoders, i.e., Dj and Dp. with the same structure for both the holistic and partial branches. During training, the label interrupted by Gaussian noise, Yr, is used as input to the decoder together with the spatial, temporal, and frequency embedding conditions as shown in the following equation.
S (y,,,, 22). Dp(y,.
where the predicted action segmentation results from those two branches are merged together during the inference time. The outputs are supervised directly by the original action segmentation annotations using binary cross-entropy loss and temporal boundary loss, which is used to align the action boundaries in the denoised sequence and the ground truth sequence.
5
Experiments
5.1
Implementation Details
All the experiments are conducted on 1 NVIDIA A100 GPU and PyTorch 2.0.0. We adopt both BLIP-2 [29] and CLIP [46] as the feature extractors, for all the experiment tables shown in the main paper, we set frame length as 3, 000 for each sample to get the train/val/test sets, where cross-movie evaluation denotes that the videos used for the aforementioned three sets come from different movies, which measures both of the cross- crbse-subject generalizability and cross-scenarc generalizability. On the other hand, the random partition denotes that the samples of the train/val/test sets randomly selected. For the cross-mos movie evaluation settings, the train/val/test sets consist of 566, 61, and 163 long-video samples, respectively. For the random evaluation settings, the train/val/test sets consist of 626, 82, and 82 long-video samples, respectively. Each sample has a time length of 3.3 minutes. The input feature dimension of the encoder is 768 for BLIP-2 [29] feature extractor and is 512 for CLIP [46] feature extractor, where we use ASFormer (12 layers with 256 feature maps) as the feature encoder and ASFormer (8 layers with 128 feature maps) according to [60, 34]. All the models are trained using the Adam [9] optimizer with a batch size of 4. The learning rate to train our model is selected le 4. The binary cross-entropy loss and the boundary loss are equally weighted. All the baselines in this paper adopt holistic input apart from the experiments reported in Table 6.
In line with prior studies in the field of human action segmentation [34, 40, 21, we evaluate model performance using frame-wise accuracy (ACC), edit score (EDIT), and segmental Fl scores at overlap thresholds (10%, 25%, 50%) (F1 @ 10, 25, 50). The ACC metric captures the accuracy of the predictions at the individual frame level, whereas the EDIT and F1 scores focus on the temporal consistency and quality of action segmentation between segments. Note that, approach is two-stage approach, where we rely on pre-extracted VLM features.
5.2 Benchmark Analysis
We first conduct training and evaluations on the random partition of the RHAS dataset and use BLIP- 2 [29] as the feature extractor, where the experimental results are shown in Table 2. The referring action recognition approach. i.e., RefAtomNet [42]. shows the second-best ACC, i.e., 39.48%, while its performances of F1 @ [10, 25, 50} are limited, since it is not specifically designed for long video understanding and lacks long temporal reasoning capability. Most of the human action segmentation baselines delivers better Fl @ (10, 25, 50), where L.TContent and ActDiff achieve > 60% of F1 @ (10, 25, 50), thanks to their promising long-term dependency modeling capability. Among all the human action segmentation basclines, ActDiff [34] shows best performance for the referring human action
7

---

Table 2:
Experimental results with BLIP-2 [29] cross-modal feature extractor and under random partition setting on the RHAS dataset.
Val
Test
Method
ACC
EDIT
F1
@
F1
F1@55
ACC
EDIT
F1@ 10
F1 25
F1@50
FACT [40]
26.30
0.27
55.86
54.06
50.38
26.08
0.27
52.91
50.77
47.06
ActDiff [34]
42.01
7.05
70.74
68.19
63.66
41.85
7.20
70.56
68.34
63.29
ASQuery [18]
31.37
0.10
35.21
33.90
31.94
32.30
0.12
35.59
33.58
29.51
LTContent [2]
35.72
68.70
66.40
62.25
34.23
0.31
64.70
63.09
58,50
RefAtomNet [42]
39.48
0.12
36.22
34.68
32.48
38.01
0.13
34.01
31.93
27.62
Ours
62.69
7.50
89.07
86.23
80.91
62.58
7.75
87.96
85.50
79.39
Table 3:
Experimental results with BLIP-2 [29] cross-modal feature extractor and under cross-movie evaluation setting the RHAS dataset.
Val
Test
Method
ACC
EDIT
F1@ 10
F1@25
F1@50
ACC
EDIT
F1@10
F1@
F1 50
FACT [40]
26.01
0.34
60.16
57.79
55.33
27.89
0.44
59.05
58.37
57.13
ActDiff [34]
4,46
5.96
38,86
38.36
37.51
2.36
15,09
22.44
22.28
21.80
ASQuery [18]
22.95
0.05
34.39
31.52
27.96
20.84
0.08
33.86
32.86
30.20
LTContent [2]
49.49
0.33
42.69
40.12
37.22
52.52
0.37
49.35
47.24
42.55
RefAtom [42]
35.00
0.10
49.58
46.60
41.58
38.73
0.13
41.16
39.17
35.44
Ours
54.41
6.31
89.14
86.56
83.35
59.63
19.37
90.91
90.33
89.26
Table 4: Experiments with CLIP [46] and under random partition on RHAS133.
Val
Test
Method
ACC
EDIT
@
F1
55
ACC
EDIT
FI@ 10
FIG
FI@50
FACT [40]
23.99
0.46
56.66
54.95
51.06
23.95
0.44
55.10
$3.63
49.72
ActDiff [34]
13.31
7.28
29.88
28.84
25.92
13.21
7.38
29.19
28.19
25.68
ASQuery [18)
32.59
0.07
29.63
28.52
26.97
26.35
0.06
25.08
23.82
19.79
LTContent [2]
13.20
2.08
35.29
30.96
28.24
13.99
1.78
30.96
29.20
22.38
NefAtomNet [42]
27.27
0.10
29.38
28.25
25.92
27.80
0.11
29.27
27.86
24.41
Ours
46,99
7.68
77.00
74.39
69.57
46.62
7.44
74.74
72.28
73.16
Table 5: Ablations on RHAS133 using BLIP-2 [29] and under partition setting on different movies.
Val
Test
Method
ACC
EDIT
F 10
FI@Z 25
F1@50
ACC
EDIT
F1@ 10
F1@ 25:2
F1@50
w/o all HCMGB
4,46
5.96
38,86
38.36
37.51
2,36
15.09
22.44
22.28
21.80
6.34
23.44
49.42
w/o HP-XLSTM
47.75
6.30
82.81
80.61
77.63
52.01
19.68
85.06
84.54
83.50
w/o BCAF
44.24
6,35
79.76
77.38
74.50
49.56
19.87
82.98
82.45
81.38
DFFT-cond
6.33
80.52
55.49
19.92
87.42
86.38
FACT [40)
26.01
0.34
$5.33
27.89
0.44
58.37
$7.13
FACT [40] w/ partial branch
18.35
0.32
62.57
61.10
61.10
18.34
0.43
63.72
63.35
62.40
ActDIFF [34]
4.46
5.96
38.86
38.36
37.51
3,26
15.09
22.44
22.28
21.80
ActDIFF [34] w/ DFF--cond
10.46
6.29
26.61
24.80
23.40
11.92
19.54
30.01
29.64
28.92
Cross-Ateetion Fusion [7]
51.74
6.41
86.91
84.47
81.50
57.02
19.87
88.97
88.41
87.36
Ours
54.41
6.31
89.14
86.56
83.35
59.63
19.37
90.91
90.33
segmentation task, which delivers 42.01% of ACC, 7.05% of EDIT, 70.74%, 68.19%, abd 63.66% of Fl @ [10, 25, 50) the validation set and 41.85% of ACC, 7.20% of EDIT, 70.56%, 68.34%, and 63.29% of F1 @ (10, 25, 50) on the test set.
ieftusion-based human action segmentation benefits from its diffusion based framework, which iteratively refines action predictions from noise, enabling to capture complex temporal dependencies and action priors, which benefits the cue reasoning of the referring human action segmentation task. However, we observe that limited performance delivered by ActDiff [34] on the cross-movie evaluation setting, as shown in Table 3. One underlying reason is that its learned action priors,

---

e.g., temporal position and relational dependencies. are subject- and svenario-specifii and may not transfer effectively to movies with different scene structures, pacing, or narrative conventions. Among all the baselines used in the benchmark, I.TContent [2] produces the best ACC as shown in Table 3, Le., 49.49%, while FACT [40] delivers best performances in terms of F1 @ [10, 25, 50), i.e., 60.16%, 57,,9, and 55.33% on the validation set and 59.05%, 58.37%, and 53%1 on the test set. Regarding the cross-movie evaluation, most of the benchmarked baselines suffer performance decays in terms of FI @ (10, 25, 50), indicating that cross-scenario and cross-subject setting is challenging for most of the human action segmentation baselines in the RHAS task, while the referring human action recognition baseline RefAtomNet [42] works better in cross-movie evaluation scenario, thanks to its cross brasc- agent attention mechanism which could will address the information fusion between local and global cues. ActDiff delivers the best EDIT score both the random partition and cross-movie partition due to its iterative denoising mechanism, which enhances temporal consistency and boundary precision across varying data distributions. Since our task focuses on fine-grained action annotations, recognizing precise action boundaries is inherently difficult, leading to lower EDIT scores.
Our proposed HopaDIFF outperforms existing diffusion- action segmentation methods by incorporating holistic-partial feature fusion through HP-xLSTM and introducing frequency-domain conditioning, which together enhance target-aware reasoning and enable more precise and controllable action segmentation in complex
scenarios. FIFaDIFF delivers 62.69% of ACC, 7.50% of EDIT, 89.07 99.07% 80.91%. 91. and 69.57% of FI @ 10. 25, 50) on the validation set and 62.58% of ACC, 7.75% of EDIT, 87 97.,,% 85.50% and 79.39% of FI @ 110 25, 50) on the test set on random partition. We can also observe that HIFaDIFF delivers similar F1 @ ( (10, 25, 50) on the cross- movie evaluation setting, which overcomes the shorteomings of the existing diffusion-bar baased human action segmentation approach by using high control granularity provided by holistic- parttcl-paaalll aware XLSTM-based temporal reasoning and the frequency condition.
We further evaluate the cross VLM backbone generalizability of all the leveraged baselines and our proposed HopaDIFF method in Table 4. We first observe that most of the leveraged approaches show better performances when they use the BLIP-2 [29] cross- mooal feature extractor compared with the CLIP [46] cross-moda feature extractor, indicating that using strong VLM backbone can benefit the RHAS task. Among all compared methods, our proposed HOFaDIFF achieves the best performance across every metric on both validation and test sets. Specifically, HopaDIFF reaches 46.99 46.99% ACC, 7.68% EDIT, and 77. 77.00%, 74.39% 69.57% 57% F1 scores on the validation set, and 46. 66.62% ACC, 7.44% EDIT, and 74.74%, 72.28%, 73. 76% F1 scores the test set. These results surpass all baselines by large margins. particularly in temporal consistency and segmental precision. In contrast, the best baseline FACT only achieves 23.99% ACC and maximum Fl @ 50 of 49. 19.72% on the test set, indicating the difficulty of using weaker VLM backbone in our RHAS task.
Notably, HoFaDIFF maintains stable performance across diverse evaluation settings, highlighting its generalization capability. This performance gain is attributed to its branch design that fuses holistic and partial cues, and the integration of Fourier-basee frequency conditioning for finer generative control. Overall, the strong results confirm that HopaDIFF effectively addresses the unique challenges of referring human action segmentation in complex, multi-person scenarios. We further provided the number of trainable parameters, inference times, and GFLOPS of baselines and our HopaDIFF in Table 7 for efficiency comparison.
5.3 Ablation Analysis
The ablated module variants include: (1) w/o all, a vanilla diffusion-based action segmentation model without any of the proposed components; (2) w/o HCMGB, which removes the GroundingDINO [37]- based partial reasoning branch; (3) w/o HP-xLSTM, which omits the holistic-partic aware XLSTM temporal aggregation module; (4) w/o BCAF, which excludes the bidirectional cross-attention fusion between the input gates of the two branches; and (5) w/o DFT-cond, which eliminates frequency- domain conditioning in both branches. The full HopaDIFF model achieves the highest performance, with 59.63% ACC, 19.37% EDIT, and 90.91%, 90.33%, 89.26% F1 scores at thresholds 10%. 25%, and 50% on the test set. Removing HP-XLSTM leads to a significant drop in F1 @ 50 from 89. 89.26% to 83.50% demonstrating the importance of long-range holistic-particn temporal modeling. Eliminating DFT-cond lowers Fl 50 to 86.38% validating the effectiveness of frequency- conditioning in improving the precision and controllability of action segmentation generation. The absence of BCAF

---

Table 6: Ablation experiments for partial branch, when use BLIP-2 and cross-movie partition.
Val
Test
Method
ACC
EDIT
FI@10 10
F1@25
F1@50
ACC
EDIT
F1 @ 10
F1@25
F1@50 50
FACT [40]
16.73
0.39
66.07
65.73
64.69
15.82
0.30
57.30
55.75
53.72
ActDIFF [34]
3.84
3.51
51.38
50.53
47.87
2,04
7.67
44.13
43,70
42.42
ASQuery [18]
32.42
0.11
51.18
48.31
43.32
36.51
0.12
44.46
42.52
39.09
LTContent [2]
45.36
0.08
60.51
57.00
50.87
50.34
0.08
49.58
47.14
42.43
RefAtomNet [42]
27.73
0.13
46.90
43.52
38.82
30.45
0.15
38.01
36.30
32.72
Ours w/ only PB
44.48
6,28
79.40
77.34
74,42
48.45
19.70
81.99
81.49
80.48
Ours w/ HPB
54.41
6.31
89.14
86,56
83.35
59.63
19.37
90.11
90.33
89.26
also causes a performance decrease, with F1 @@ dropping to 81.38%, highlighting the utility of cross- branch interaction through input gate attention. Notably, removing the GroundingDINO [37]-based partial reasoning branch (HCMGB) reduces Fl @ 50 to 49.42%, showing that person-specific local cues are essential for accurate target-aware segmentation. Overall, the ablation results confirm that each proposed component, i.e., GroundingDINO [37]-based partial cue reasoning. HP-xxSTMM-bar fusion, and frequency conditioning, plays critical role in our model, and their synergy leads to the superior performance of HopaDIFF on the complex textual referring action segmentation task. The experiments also reveal that replacing the action branch with the partial branch FACT ((6) FACT [40] w/ parrial branch) improves F1 @ (10,25,50] by better capturing long-duration, movement- related actions, but decreases ACC due to the loss of broader contextual cues. Adding frequency- domain condition to ActDIFF ((7) ActDIFF [34] w/ DFT-cond) enhances boundary detection (higher EDIT) but shows weaker generalization across movies without the HP-xLSTM module, leading to lower Fl on validation. Finally, while cross-attention fusion performs competitively, our HP- XLSTM achieves superior accuracy and Fl by enabling structured, fine ame-grained temporal modeling and selective bidirectional information exchange between holistic and partial branches. We further test cropping the person region detected by GroundingDino before feeding it into the baselines (Table 6). Compared to holistic inputs (Table 3), this results in lower ACC but partial gains in F1, showing the complementary roles of holistic and partial branches: the holistic branch captures background and interaction cues, while the partial branch focuses on the target individual to improve movement-related action recognition.
Conclusion
In this work, we introduced Referring Human Action Segmentation (RHAS), new task aimed at segmenting actions of individuals in multi-person untrimmed videos using natural language references. To support this, we constructed the RHAS133 dataset with fine-grained action labels and referring expressions. We proposed HIpaDIFF, a dual-branch diffusion model that fuses global and localized cues via HP-xLSTM and DFT-based conditioning for improved temporal reasoning and control. Extensive experiments show that HopaDIFF outperforms existing baselines in both accuracy and temporal consistency. Our contributions pave the way for fine grained language-guie -uided video understanding in complex settings.
Acknowledgements
The project is funded by the Deutsche Forsehungsgemeiinccaaft (DFG, German Research Foundation) -SFB 1574 471667778 This work was also partially supported by the SmartAge project sponsored by the Carl Zeiss Stiftung (P2019-01-003; 2021-2026). This work was performed on the HoreKa supercomputer funded by the Ministry of Science. Research and the Arts Baden-Württenn Württemberg and by the Federal Ministry of Education and Research. The authors also acknowledge support by the state of Baden- Württemberg through bwHPC and the German Research Foundation (DFG) through grant INST 35/1597-1 FUGG. This project is also supported in part by the National Natural Science Foundation of China under Grant No. 62473139, in part by the Hunan Provincial Research and Development Project (Grant No. 2025QK3019), and part by the Open Research Project the State Key Laboratory of Industrial Control Technology, China (Grant No. ICT2025B20). This research was partially funded by the Ministry of Education and Science of Bulgaria (support for INSAIT, part of the Bulgarian National Roadmap for Research Infrastructure).
10

---

References
[1] Nicolas Aziere and Sinisa Todorovic. Multistage temporal convolution transformer for action segmentation. Image and Vision Computing, 2022.
[2] Emad Bahrami, Gianpiero Francesca, and Juergen Gall. How much temporal long-term context needed for action segmentation? In ICCV, 2023.
[3] Siddhant Bansal, Chetan Arora, and C. V. Jawahar. My view is the best view: Procedure learning from egocentric videos. In ECCV, 2022.
[4] Maximilian Beck, Korbinian Pöppel, Markus Spanring, Andreas Auer, Oleksandra Prudnikova, Michael Kopp, Günter Klambauer, Johannes Brandstetter, and Sepp Hochreiter. xLSTM: Extended long short-term memory. In NeurIPS, 2024.
[5] Yuqi Bu, Liuwu Li, Jiayuan Xie, Qiong Liu, Yi Cai, Qingbao Huang, and Qing Li. Scene-text oriented referring expression comprehension. TMM, 2023.
[6] Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? A new model and the Kinetics dataset. In CVPR, 2017.
[7] Chun-Fu Richard Chen, Quanfu Fan, and Rameswar Panda. Cross VIT: Cross-attention multi- scale vision transformer for image classification. ICCV, 2021.
[8] Ronghao Dang. Jiangyan Feng, Haodong Zhang, Chongjian Ge, Lin Song, Lijun Gong, Chengju Liu, Qijun Chen, Feng Zhu, Rui Zhao, and Yibing Song. InstructDET: Diversifying referring object detection with generalized instructions. In ICLR, 2024.
[9] Kingma Diederik. Adam: A method for stochastic optimization. (No Title), 2014.
[10] Guodong Ding, Fadime Sener, and Angela Yao. Temporal action segmentation: An analysis of modern technique. arXiv preprint arXiv:2210.100:: 2022.
[11] Zihan Ding. Tianrui Hui, Junshi Huang, Xiaoming Wei, Jizhong Han, and Si Liu. Language- bridged spatial-temee temporal interaction for referring video object segmentation. In CVPR, 2022.
[12] Dazhao Du, Bing Su, Yu Li, Zhongang Qi, Lingyu Si, and Ying Shan. Do we really need temporal convolutions in action segmentation? arXiv preprint arXiv:2205, 13425, 2022.
[13] Zexing Du and Qing Wang. Dilated transformer with feature aggregation module for action segmentation. Neural Processing Letters, 2022.
[14] Yazan Abu Farha and Jurgen Gall. MS-TCN: Multi-stage temporal convolutional network for action segmentation. CVPR, 2019.
[15] Alireza Fathi, Xiaofeng Ren, and James M. Rehg. Learning to recognize objects in egocentric activities. In CVPR. 2011.
[16] Yuqian Fu, Chengrong Wang, Yanwei Fu, Yu-Xiong Wang, Cong Bai, Xiangyang Xue, and Yu-Gang Jiang. Embodied one-shot video recognition: Learning from actions of a virtual embodied agent. In ACMMM, 2019.
[17] Yuqian Fu, Runze Wang, Yanwei Fu, Danda Pani Paudel, Xuanjing Huang, and Luc Van Gool. ObjectRelator: Enabling cross-view object relation understanding in ego-centrie and exo-centric videos. In ICCV, 2025.
[18) Ziliang Gan, Lei Jin, Lei Nie, Zheng Wang, Li Zhou, Liang Li, Zhecan Wang. Jianshu Li, Junliang Xing, and Jian Zhao. ASQuery: A query-based model for action segmentation. In ICME, 2024.
[19] Shang-Hua Gao, Qi Han, Zhong-Yo Li, Pai Peng, Liang Wang, and Ming-Ming Cheng. Global2Local: Efficient structure search for video action segmentation. CVPR, 2021.
[20] Yue Gao, Jiaxuan Lu, Siqi Li, Yipeng Li, and Shaoyi Du. Hypergraph-based multi-vier action recognition using event cameras. TPAMI, 2024.

---

[21] Chunhui Gu, Chen Sun, David A. Ross, Carl Vondrick, Caroline Pantofaru, Yeqing Li, Sudheen- dra Vijayanarasimhan, George Toderici, Susanna Ricco, Rahul Sukthankar, Cordelia Schmid, and Jitendra Malik. AVA: A video dataset of spatio-temporally localized atomic visual actions.
In CVPR, 2018.
[22] Basavaraj Hampiholi, Christian Jarvers, Wolfgang Mader, and Heiko Neumann. Depthwise separable temporal convolutional network for action segmentation. In 3DV, 2020.
[23] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NeurlPS, 2020.
[24] Shaofei Huang, Rui Ling, Hongyu Li, Tianrui Hui, Zongheng Tang, Xiaoming Wei, Jizhong Han, and Si Liu. Unleashing the cemporal-ppatinn reasoning capacity of GPT for training- free audio and language referenced video object segmentation. In AAAI, 2025.
[25] Yifei Huang, Yusuke Sugano, and Yoichi Sato. Improving action segmentation via graph-based temporal reasoning. CVPR, 2020.
[26] Anna Khoreva, Anna Rohrbach, and Bernt Schiele. Video object segmentation with language referring expressions. In ACCV, 2019.
[27] Hilde Kuehne, Ali Arslan, and Thomas Serre. The language of actions: Recovering the syntax and semantics of goal-directed human activities. In CVPR, 2014.
[28] Colin Lea, Michael D. Flynn, René Vidal, Austin Reiter, and Gregory D. Hager. Temporal convolutional networks for action segmentation and detection. In CVPR, 2017.
[29] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. BLIP-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In ICML, 2023.
[30] Ruiyu Li, Kaican Li, Yi-Chun Kuo, Michelle Shu, Xiaojuan Qi, Xiaoyong Shen, and Jiaya Jia. Referring image segmentation via recurrent refinement networks. In CVPR, 2018.
[31] Shi-Jie Li, Yazan AbuFarha, Yun Liu, Ming-Ming Cheng, and Juergen Gall. MS-TCN++: Multi-stage temporal convolutional network for action segmentation. TPAMI, 2023.
[32] Yunheng Li, Zhuben Dong, Kaiyuan Liu, Lin Feng, Lianyu Hu, Jie Zhu, Li Xu, Shenglan Liu, et al. Efficient two-step networks for temporal action segmentation. Neurocomputingg 2021.
[33] Jiacheng Lin, Jiajun Chen, Kunyu Peng. Xuan He, Zhiyong Li, Rainer Stiefelhagen, and Kailun Yang. EchoTrack: Auditory referring multi-object tracking for autonomous driving. T-ITS, 2024.
[34] Daochang Liu, Qiyue Li, Anh-Dung Dinh, Tingting Jiang, Mubarak Shah, and Chang Xu. Diffusion action segmentation. In ICCV, 2023.
[35] Jinjie Liu, Guolong Liu, Huan Zhao, Junhua Zhao, Jing Qiu, and Zhao Yang Dong. Real- time industrial carbon emission estimation with deep learning-based device recognition and incomplete smart meter data. EAAI, 2024.
[36] Runtao Liu, Chenxi Liu, Yutong Bai, and Alan L Yuille. CLEVR-Reft: Diagnosing visual reasoning with referring expressions. CVPR, 2019.
[37] Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang. Qing Jiang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, and Lei Zhang. Grounding DINO: Marrying DINO with grounded pre-training for open-set object detection. In ECCV, 2024.
[38] Yuanzhong Liu, Junsong Yuan, and Zhigang Tu. Motion-driven visual tempo learning for video-based action recognition. TIP, 2022.
[39] Zhichao Liu, Leshan Wang, Desen Zhou, Jian Wang, Songyang Zhang, Yang Bai, Errui Ding, and Rui Fan. Temporal segment transformer for action segmentation. arXiv preprint 2rXiv:2302.1304, 2023.
12

---

[40] Zijia Lu and Ehsan Elhamifar. FACT: Frame-action aross-attention temporal modeling for efficient action segmentation. In CVPR, 2024.
[41] Junyong Park, Daekyum Kim, Sejoon Huh, and Sungho Jo. Maximization and restoration: Action segmentation through dilation passing and temporal reconstruction. Patlern Recognition, 2022.
[42] Kunyu Peng, Jia Fu, Kailun Yang, Di Wen, Yufan Chen, Ruiping Liu, Junwei Zheng, Jiaming Zhang, M. Saquib Sarfraz, Rainer Stiefelhagen, and Alina Roitberg. Referring atomic video action recognition. In ECCV, 2024.
[43] Kunyu Peng, Alina Roitberg, Kailun Yang, Jiaming Zhang, and Rainer Stiefelhagen. Should I take walk? Estimating energy expenditure from video data. In CVPRW, 2022.
[44] Kunyu Peng,
Alina Roitberg, Kailun Yang, Jiaming Zhang, and Rainer Stiefelhagen. Trans- DARC: Transformer-basel driver activity recognition with latent space feature calibration. In IROS, 2022.
[45] Kunyu Peng, Cheng Yin, Junwei Zheng, Ruiping Liu, David Schneider, Jiaming Zhang, Kailun Yang. M. Saquib Sarfraz, Rainer Stiefelhagen, and Alina Roitberg. Navigating open set scenarios for skeleton-based action recognition. In AAAI, 2024.
[46] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In JCML, 2021.
[47] Fadime Sener, Dibyadip Chatterjee, Daniel Shelepov, Kun He, Dipika Singhania, Robert Wang, and Angela Yao. Assembly 101: large-scale multi-view video dataset for understanding procedural activities. In CVPR, 2022.
[48] Seonguk Seo, Coon-Young Lee, and Bohyung Han. URVOS: Unified referring video object segmentation network with a large-scale benchmark. In ECCV, 2020.
[49] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. ICLR, 2021.
[50] Sebastian Stein and Stephen J. McKenna. Combining embedded accelerometers with computer vision for recognizing food preparation activities. In UbiComp, 2013.
[51] Xiaoyan Tian, Ye Jin, and Xianglong Tang. Local-global transformer neural network for temporal action segmentation. Multimedia Systems, 2022.
[52] Arun Balajee Vasudevan, Dengxin Dai, and Luc Van Gool. Object referring in videos with language and human gaze. In CVPR, 2018.
[53] Sarvesh Vishwakarma and Anupam Agrawal. A survey on activity recognition and behavior understanding in video surveillance. The Visual Computer, 2013.
[54] Dong Wang, Yuan Yuan, and Qi Wang. Gated forward refinement network for action segmenta- tion. Neurocomputing, 2020.
[55] Jiahui Wang, Zhenyou Wang, Shanna Zhuang, and Hui Wang. Cross-cnhancement transformer for action segmentation. arXiv preprint XXIv:2205.0044/ 2022.
[56] Jue Wang and Lorenzo Torresani. Deformable video transformer. In CVPR, 2022.
[57] Zhenzhi Wang, Ziteng Gao, Limin Wang, Zhifeng Li, and Gangshan Wu. Boundary cascade networks for temporal action segmentation. In ECCV, 2020.
[58] Dongming Wu, Wencheng Han, Tiancai Wang, Xingping Dong, Xiangyu Zhang, and Jianbing Shen. Referring multi-object tracking. In CVPR, 2023.
[59] Jiannan Wu, Yi Jiang, Peize Sun, Zehuan Yuan, and Ping Luo. Language as queries for referring video object segmentation. In CVPR, 2022.
13

---

[60] Fangqiu Yi, Hongyu Wen, and Tingting Jiang. ASFormer: Transformer for action segmentation. In BMVC, 2021.
[61] Licheng Yu, Patrick Poirson, Shan Yang, Alexander C. Berg, and Tamara L. Berg. Modeling context in referring expressions. In ECCV, 2016.
[62] Junbin Zhang. Pei-Hsuan Tsai, and Meng-Hsun Tsai. Semantic2Graph: Graph-based multi- modal feature for action segmentation in videos. arXiv preprint arXiv:2209.0565.. 2022.
[63] Yilun Zhang. Yuqian Fu, Xingjun Ma, Lizhe Qi, Jingjing Chen, Zuxuan Wu, and Yu-Gang Jiang. On the importance of spatial relations for few-shot action recognition. In MM, 2023.
[64] Yunlu Zhang, Keyan Ren, Chun Zhang, and Tong Yan. SG-TCN: Semantic guidance temporal convolutional network for action segmentation. In IJCNN, 2022.
[65] Zixin Zhu, Xuelu Feng, Dongdong Chen, Junsong Yuan, Chumming Qiao, and Gang Hua. Exploring pre-trained text-to-video diffusion models for referring video object segmentation. In ECCV, 2024.
14

---

Appendix
A Society Impact and Limitations
A.1 Society Impact
In this work, we introduce Referring Human Action Segmentation (RHAS), novel task that enables fine-grained action understanding in complex, multi-person video scenes using natural language descriptions. We contribute the first dataset for this task, RHAS133, along with diffusion-batte based referring human action segmentation solution, HopaDIFF, which integrates global and localized cues to achieve precise, target-aware action segmentation. These contributions lay important groundwork for the future of language-guide video understanding.
This research has the potential to significantly impact several real-world domains. In healthcare and eldercare, RHAS can enable systems to achieve action segmentation shared spaces with multiple persons when textual reference is given for specific person, e.g., "the elderly man in the red shirt". In human-robot interaction, robots could better understand and track human behaviors in dynamic group settings. Similarly, in surveillance and interpret behaviors based on spoken or written descriptions, offering more intuitive and flexible interaction with video analytics tools.
safety monitoring, RHAS may help systems localize
However, this technology also raises ethical considerations. First, the reliance on data from publicly available movies may embed social, cultural, or gender biases present in media, which could affect model fairness and generalizability. For instance, actions associated with underrepresented groups may be poorly modeled, leading to biased or inaccurate outputs. Second, RHAS systems might be misused for invasive surveillance or profiling if deployed without safeguards, raising privacy and civil liberty concerns.
To mitigate these risks, future work should focus on curating more diverse, representative datasets and embedding fairness-aware training objectives. Transparency, stakeholder oversight, and responsible deployment frameworks will be critical ensuring that the benefits of RHAS technologies are equitably distributed and ethically aligned with societal values.
A.1.1 Limitations
As stated in our main paper, our work is the first to investigate human action segmentation guided by textual references enable action segmentation for specific individual in multi-person scenarios. Due to the novelty of the task, experiments are currently limited to our constructed dataset, which constrains the evaluation of the proposed model's generalizability. Nevertheless, we made substantial efforts to ensure dataset diversity, collecting data from a wide range of movies over an annotation process that spanned more than 8 months. To further assess generalizability, evaluation protocols, e.g., random partition and cross-movie partition, and conducted cross-baa tross-backbone experiments by replacing BLIP-2 [29] with CLIP. future work, we aim to expand the dataset by incorporating larger and more diverse set of movies. Our method is two-stage method that relies on pre-extracted VLM features, limiting the efficiency of inference. However, we also note that most existing methods (e.g., FACT, ActDIFF, I.TContext, ASQuery) also adopt a two-stage processing pipeline due to the high computational cost of long video processing. We acknowledge this point also as an open challenge in action segmentation field.
designed multiple
B Further Clarification of the BCA Mechanism
The BCA module enables information exchange between holistic and partial branches by using cross-input-gate attention to modulate input information for each stream jointly, supporting richer temporal reasoning. HP-xLSTM leverages cross-input gated attention (i.e., BCA applied input gates), enabling each branch to selectively incorporate temporal cues from the other based on feature- level gating. This design enhances temporal segmentation and boosts per-frame accuracy, particularly in challenging multi-person scenarios with fine-grain fine-grained actions.
15

---

The attention weights and fused representations are computed as follows:
(WO vd
a1
H-P = softmax
(8)
X =t H-P. -WWx ( Vd
(9)
a
P-H softmax
(10)
X =
0 of XWvX;";
(11)
where Wq. Wx, Wy E Iddd are learnable projection matrices for query, key, and value. X," and X are the cross-attended inputs xLSTM input gates the corresponding branches.
C More Implementation Details
In this section, we will provide more implementation details regarding our proposed HopaDIFF framework. For both encoders the holistic branch and the partial branch, the number of feature maps is chosen as 64, and the kernel size is chosen as 5.
We adopt normal dropout rate of 0.5, channel dropout rate of 0.5, and a tem- poral dropout rate of 0.5. For both the decoders of the holistic branch and the partial branch, we choose the dimension of time embedding as 512, the number of feature maps as 24, the kernel size as 5, and the dropout rate as 0.1. Regarding the hyperparameters of the diffusion pro- choose the timesteps 1, 0,000
Table 7: Efficiency measurement when we leverage BLIP- 2 feature extractor on RHASI33, cross-movie partition.
Method
Trainable Param.
Inference Time
GFLOPS
FAC.T [40]
10,20M
1.985
16.37Ci
ACTDIFE [34]
ASQuery [18]
47,28M
1.24s
57.22G
I.TContent [2]
6.12M
2,70c
20,020
RefAtomNet
273.53G
HapaDIFF
20M
2,81s
10.72G
and the sampling time steps as 25. The training epoch is set as 1, 000. The number of parameters,
inference time, and GFLOPS of the baseline approaches and our approach reported in Table 7.
D Ablation Study of the frame length
In this subsection, we provide further ablation regarding the frame length for the data partition, where change the setting used in the main paper from 3. 000 2, 000, where the results demonstrated in Table We could observe that our proposed HopaDIFF outperforms the others with large margins under this frame length setting due its superior capability to achieve highly controllable human action segmentation generation. This ablation further illustrates the superior generalization capability of HOPaDIFF model.
Table 8: Experimental results with BLIPv2 cross- modal feature extractor and under cross-movie evaluation setting on the RHAS dataset, using frame length 2000.
Val
Test
Method
ACC
EDIT
FI@
F1
F0@550
ACC
EDIT
F1@ 10
FI 25
F1@50
FACT [40]
38.05
0.44
75.26
73.88
70.52
36.95
0.50
73.89
72.84
70.62
ActDiff [34]
4.59
25.48
38.96
38.59
37.66
3.00
54,96
27.41
27.08
26.14
ASQuery [18]
36.12
0.12
31.40
29.70
25.91
34.12
0.12
35.24
33.24
28.25
LTContent [2]
12.80
0.55
29.12
27.88
24.40
13.65
0.59
31.18
29.18
25.15
Net [42]
29.47
0.12
32.80
31.00
27.14
33.62
0.15
46.29
44.13
39.32
Ours
59.60
35.00
94.33
92.62
88.85
62.63
90.78
94.71
93.57
91.19
E
Ablation regarding the Dual Head Setting
Training two branches separately leads better performance compared with the mentioned coun- terpart, shown in Table 9. It presents an ablation study comparing the Merged bascline with
16

---

our dual-head design. On the validation set, our approach achieves higher accuracy (54.41% vs. 52.12%) and consistently outperforms across all Fl metrics, with improvements of +2.04%, +2.05%. and +1.66% at F1 @ 10, F1 @25, and Fl @ 50, respectively. On the test set, our method similarly shows superior accuracy 159.63% vs. 56.50%) and notable gains FI scores across all thresholds. Although the merged setting achieves slightly higher EDIT (6.42 vs. 6.31 on validation, 19.89 vs. 19.37 on test), our dual head design provides a better balance, enhancing both per-frame accuracy and temporal overlap performance. These results highlight the advantage of explicitly modeling holistic and partial branches separately rather than merging them, as the dual head structure allows more fine-gra fine-grained temporal reasoning and better recognition of actions across different contexts.
Table Ablation study of the dual head setting.
Val
Test
Method
ACC
EDIT
@1@10 10
F1@2 25
F1 @105
ACC 56.50
EDIT
F1@10
F1
F1@50 50
Merged Ours
52.12
6.42
87.10
84.51
81.49 83.35
19.89
88.28
87.69
86.64
54.41
6.31
89.14
86.56
59.63
19.37
90.11
90.33
89.26
F Ablation regarding the Diffusion Action Segmentation Mechanism
To evaluate the contribution of the proposed diffusion mechanism in human action segmentation framework, we conduct an ablation study by removing the diffusion module and comparing the results against our full model. The quantitative results are reported in Table 10, covering both validation and test sets across multiple metrics.
Without the diffusion mechanism, the model demonstrates clear degradation in performance across all metrics. On the validation set, the accuracy drops from 54.41% to 53.43%, while the EDIT score declines drastically from 6.31 0.09, indicating poor temporal alignment. Similarly, Fl scores decrease significantly, e.g., from 89.1 19.14% to 67. 67.35% at Fl @ 10 and from 83.35% to 81.50% at F1 @50 similar trend is observed the test set, where the absence of diffusion reduces the EDIT score from 19.37 to 0.10, and segmental Fl scores experience large drops, such as from 90.33% to 56. 17% Fl @25.
These results highlight the importance of the diffusion mechanism for our model. The large improve- ment in the EDIT score shows that diffusion plays a critical role in refining temporal boundaries and improving sequence coherence. Furthermore, the consistent gains across all F1 metrics confirm that diffusion enhances the model's ability capture fine- greigra action transitions, ultimately leading to more robust and accurate action segmentation.
Table 10: Ablation study of the diffusion human action segmentation mechanism.
Val
Test
Method
ACC
EDIT
FI@ 10
F1@25
F1@50
ACC
EDIT
FI@ 10
FI 25
F1 @5@
w/o Diffusion
53.43
0.09
67.35 89.14
63.68
81.50
57.47
58.56 90.11
56.17 90.33
53.72
Ours
54.41
6.31
86.56
83.35
59.63
19.37
89.26
G Ablation regarding the Efficacy of Different Branches
To further investigate the contribution of each component in our framework, we conduct an ablation study the partial and holistic branches. The results are summarized in Table 11, where we compare the performance of our full model against two variants: one without the partial branch and one without the holistic branch.
When the partial branch removed, the performance drops drastically across all metrics. On the validation set, the accuracy decreases from 54.41% % to 20.78%, and F1 scores suffer severe @50. A
degradation, egg., from 89. 14% to 45.81% F1 @ 10 and from 83.35% to 41 1.5%% similar trend is observed on the test set, where accuracy falls from 59.63% to 23, 44%, and F1@25
17

---

140
120
100
80
60
40
20
13
17
21
25
29
38
37
41
45
53
5
11
65
69
73
77
84
85
89
93
97
101
103
109
113
117
121
125
129
Figure 4: An overview of the statistics regarding the number of persons per movie in our RHAS133
dataset. The horizontal axis denotes the video ID, and the vertical axis denotes the number of persons annotated in the corresponding video.
decreases sharply from 90.33% to 54...%. These results highlight the critical role of the partial branch in capturing fine-grained and localized action cues.
Removing the holistic branch also results in performance degradation, though the drop is less severe than removing the partial branch. On the validation accuracy decreases 44.48%, while F1 @ 10 and FI @ 25 decline to 79.40% and 77.3 77.34%, respectively. On the test set, the accuracy reduces to 48.45%. with F1 @25 dropping from 90.33% to 81.49%. This indicates that the holistic branch contributes to capturing global temporal dependencies and overall action context, thereby improving segmentation coherence.
Overall, these ablations demonstrate that both the partial and holistic branches play complementary roles in method. The partial branch is essential for modeling detailed action transitions, while the holistic branch ensures global consistency. Their combination enables our model to achieve state-of- tiit--of-theaat segmentation performance.
H Qualitative Results
We further provide a visualization of qualitative results of our approach and the FACT model based on one sample from the RHAS133 dataset in Figure 5. We observe that HopaDIFF consistently produces action segmentation outputs that more closely align with the ground truth labels. Notably, our method demonstrates superior performance handling periodic action transitions along the temporal axis. This improvement can be attributed to the high granularity of the condition offered by the Fourier embeddings, especially on periodic temporal patterns, and the enhanced temporal reasoning capabilities of our proposed holistic-partial pitisticpprtiillawaree tross-input-gad xLSTM architecture for different visual focuses. We also identified performance drop classes with limited training samples. In such long-tail scenarios, frames corresponding to rare classes are often misclassified as more frequent classes. We acknowledge this limitation and consider tackling the long-tail distribution challenge in referring human action segmentation promising and important direction for future research.
Table 11: Ablation study of different branches in our method.
Val
Test
Method
ACC
EDIT
F1@ 10
F1@225
F1
ACC
EDIT
F1@10
F1:225 25
F1@:0
w/o partial branch
20.78
6.34
45.81
43.93
41.95
23.44
19.93
50.60
50.14
49.42
w/o holistic branch
44.48
6.28
79.40
77.34
74.42
48.45
19.70
81.99
81.49
80.48
Ours
54.41
6.31
89.14
86.56
83.35
59.63
19.37
90.11
90.33
89.26
18

---

GT
FACT
Ours
False Prediction
Correct Prediction
Figure Qualitative results of our HopaDIFF and FACT baseline, where false predictions marked as green and correct predictions are marked as blue. Each color shown in GT denotes different set of combinations of atomic-level actions.
140
120
100
80
60
40
20
13
17
25
29
33
37
45
49
53
57
61
65
73
81
85
89
93
97
101
105
109
113
117
121
125
129
133
Figure 6: An overview of the statistics regarding the number of persons per movie our RHAS133
dataset. The horizontal axis denotes the video ID, and the vertical axis denotes the number of persons annotated in the corresponding video.
I More Details of the RHAS133 Dataset
In this section, we deliver more details of our contributed RHAS133 dataset. First, we present the number of persons statistic for each movie in Fig. 6, which illustrates that the distribution of our dataset is diverse in terms of the number persons per video.
19

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The main claims made in the abstract and introduction is to explore the challenging referring human action segmentation problem by delivering new dataset and new method. Thereby it reflects the paper's 's contributions and scope.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We have illustrated the limitation of this work at the end of the conclusion
section.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and prool's
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
20

---

Answer:
[NA]
Justification:
There is no theoretical result.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We provided the details in the implementation details section of our main paper.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
21

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
Anonymous link is provided in the supplementary material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https: //wps/ips../ cc/
publ / guides/ CodeSubom ss onPoll icyy for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: cttpp.cccppuulcc///ecccc htte//ubiccgguuuii
/nipss cc/
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anodized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: is described the implement details and provided in the code.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification:
We reported the mAP, EDIT, F1 F10(10,2)) 20, 50) evaluations.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
22

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative emrr rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
They are illustrated in the implementation details.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: httpp/////////ps..eee/ http://pp.p.pco.ooeocccccccccce.
Answer: [Yes]
Justification:
We followed the guideline.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
They are described in the supplementary details.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
23

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We ensure that we cited and mentioned all the leveraged existing assets.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms of use the package should be provided. For popular datasets, papers' aw. thcode com data ses has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The new assets introduced the paper is well described in the paper, supple- mentary materials, and code.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodll as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Question:
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
25

---

16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Answer: [Yes]
Justification: We use LLM for editing, which is marked in the paper registration information.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer our LLM policy (https: / //nuurrrsss... cc/ Conferences / 2025/ LLM) for what should or should not described.
26


Reviews:
[{"review_id": "vbn1wiq3d5", "reviewer": "Reviewer_YL35", "summary": "This paper introduces **Referring Human Action Segmentation (RHAS)**, a novel task that focuses on segmenting the actions of specific individuals in multi-person untrimmed videos based on textual references. To support this task, the authors present **RHAS133**, a new dataset comprising 133 movies and 542 annotated individuals. They also propose **HopaDIFF**, a diffusion-based framework that integrates holistic and partial visual reasoning with Fourier-based conditioning to enhance action segmentation performance.", "questions": "Please refer to **Strengths And Weaknesses**.", "limitations": "Please refer to **Strengths And Weaknesses**.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "### **Strengths**\n\n1. **Novel and Well-Motivated Task**: The RHAS task addresses a real and underexplored limitation in existing action segmentation methods—namely, the over-reliance on single-person scenarios with fixed protocols. The integration of referring expressions into multi-person action segmentation is novel and practically relevant, making the problem both original and impactful.\n\n2. **Valuable Dataset Contribution**: The RHAS133 dataset is, to the best of our knowledge, the first specifically designed for referring human action segmentation. It provides rich annotations including 137 fine-grained action classes, natural language referring expressions, and multi-label annotations, making it a valuable and reusable benchmark for the community.\n\n3. **Reasonable Technical Approach**: The proposed dual-branch (holistic-partial) architecture is well-motivated, aiming to jointly capture global contextual cues and target-specific fine-grained motion patterns. The use of HP-xLSTM with a cross-input gating mechanism provides a sensible way to facilitate information exchange between branches.\n\n4. **Thorough Experimental Evaluation**: The experimental section is comprehensive, including a wide range of adapted baselines from both action segmentation and referring action recognition literature. In addition, the evaluation under both random partition and cross-movie splits offers useful insights into model generalization across settings.\n\n---\n\n### **Weaknesses**\n\n1. **Limited Technical Novelty**: The overall architecture mainly combines existing components—such as diffusion models, xLSTM, and DFT-based conditioning—rather than proposing fundamentally new algorithmic techniques. As such, the contribution is more integrative than innovative.\n\n2. **Technical and Methodological Concerns**: The model involves several components (dual branches, HP-xLSTM, BCA, and DFT conditioning), but the paper lacks sufficient justification or ablation for each design choice. Furthermore, there is no discussion or analysis of computational complexity, inference latency, or comparison with more lightweight alternatives, which raises concerns about scalability and deployment.", "was_revised": true, "final_justification": "For my part, I believe the authors have adequately addressed my main concerns. Having reviewed the other reviewers' comments, which are all positive, I am inclined to maintain my recommendation as Accept.", "rebuttal": "Dear Reviewer YL35,\n\nThank you for the review effort and recognizing the merit of our work. We have provided a point-to-point response as follows.\n\n> **W1**-Limited Technical Novelty: The overall architecture mainly combines existing components, such as diffusion models, xLSTM, and DFT-based conditioning, rather than proposing fundamentally new algorithmic techniques. As such, the contribution is more integrative than innovative.\n\nRegarding the novelty of our architecture, **we would like to clarify that our HopaDIFF is the first model to propose a cross input-gate holistic-partial aware xLSTM strategy for long-term temporal reasoning that explicitly incorporates both global video context and partial features specific to the referred individual. In addition, we introduce a novel Fourier-based condition mechanism that enhances the condition granularity by considering periodic temporal patterns from global temporal perspective in diffusion-based action segmentation.** \nTo the best of our knowledge, both of these components, cross input-gate attention within xLSTM for long term temporal reasoning between holistic and partial branches, and frequency-domain conditions are introduced for the first time for diffusion based human action segmentation.\n\nIn our experiments, we observed that existing human action segmentation methods struggle with the RHAS task, especially due to its fine-grained annotation and the complexity of multi-person, in-the-wild video scenarios. Although the diffusion-based method ActDiff achieves relatively strong performance in terms of temporal boundary modelling (e.g., high evaluation performance in terms of  EDIT score), its results vary significantly across different evaluation settings. This highlights a key challenge: the need for more expressive and fine-grained condition to support stable and accurate action segmentation prediction generation. Motivated by this, we designed the HP-xLSTM module to strengthen long-term temporal modeling, enabling bidirectional information flow between holistic and partial branches through cross input-gate attention. \nNotably, our use of cross-attention at the input gate level represents a novel contribution to xLSTM design.\nHP-xLSTM incorporates cross input-gate attention, allowing each branch to selectively integrate temporal cues from the other based on feature-level gating. This mechanism enhances temporal segmentation and improves per-frame accuracy, especially in complex multi-person scenarios involving fine-grained actions.\nComplementing this, we incorporate frequency domain information, via discrete Fourier transform to improve temporal coherence and control. Frequency conditioning, as we propose it here, allows the model to account for both low- and high-frequency patterns, which is critical for precise action segmentation in challenging real-world settings.\n\nBeyond the model itself, we would also like to emphasize our major contributions to the field. This work introduces a new task, i.e., referring to human action segmentation in multi-person scenarios, and contributes the first dataset for this purpose, RHAS133. \nThe dataset was carefully curated and annotated over the course of eight months and includes fine-grained frame-level labels guided by textual references. \nWe believe this benchmark fills an important gap in the literature and offers a solid foundation for future research in language-guided video understanding in unconstrained environments.\n\n> **W2**-Technical and Methodological Concerns: The model involves several components (dual branches, HP-xLSTM, BCA, and DFT conditioning), but the paper lacks sufficient justification or ablation for each design choice. Furthermore, there is no discussion or analysis of computational complexity, inference latency, or comparison with more lightweight alternatives, which raises concerns about scalability and deployment.\n\nWe appreciate the reviewer’s concern on the technical details and provide the response as follows.\n\n### Ablation of individual component:\nEach module, dual branches, HP-xLSTM, BCA, and DFT-based conditioning, was carefully designed to address specific challenges in referring human action segmentation, such as the need for fine-grained, temporally consistent predictions for a target individual in complex multi-person scenes and the capability to deal with periodic temporal patterns. **The effectiveness of these components is demonstrated through detailed ablation studies in Table R1-A (corresponding to Table 5 in our manuscript), where removing any of the modules leads to significant performance drops.** The ablation study evaluates five variants of our model: (1) w/o all, a baseline diffusion-based action segmentation model without any of our proposed components; (2) w/o HCMGB, which removes the GroundingDINO-based partial reasoning branch; (3) w/o HP-xLSTM, which omits the holistic-partial aware xLSTM temporal aggregation module; (4) w/o BCAF, which excludes bidirectional cross-attention between the input gates of the two branches; and (5) w/o DFT-cond, which removes frequency-domain conditioning in both branches. For example, eliminating the partial branch causes F1@50 to fall from 89.26% to 49.42%, highlighting the critical role of localized, target-aware features. \nSimilarly, excluding HP-xLSTM or the frequency-based conditioning results in consistent degradation in segmentation quality, showing that each component contributes materially to the final performance.\n\n**Table R1-A: Ablations on RHAS133 using BLIP-2 and cross-movie partition**\n\n| **Method**         | **ACC (Val)** | **EDIT (Val)** | **F1@10 (Val)** | **F1@25 (Val)** | **F1@50 (Val)** | **ACC (Test)** | **EDIT (Test)** | **F1@10 (Test)** | **F1@25 (Test)** | **F1@50 (Test)** |\n|--------------------|---------------|----------------|------------------|------------------|------------------|----------------|------------------|-------------------|-------------------|-------------------|\n| w/o all            | 4.46          | 5.96           | 38.86            | 38.36            | 37.51            | 2.36           | 15.09            | 22.44             | 22.28             | 21.80             |\n| w/o HCMGB          | 20.78         | 6.34           | 45.81            | 43.93            | 41.95            | 23.44          | **19.93**            | 50.60             | 50.14             | 49.42             |\n| w/o HP-XLSTM       | 47.75         | 6.30           | 82.81            | 80.61            | 77.63            | 52.01          | 19.68            | 85.06             | 84.54             | 83.50             |\n| w/o BCAF           | 44.24         | **6.35**       | 79.76            | 77.38            | 74.50            | 49.56          | 19.87            | 82.98             | 82.45             | 81.38             |\n| w/o DFT-cond       | 50.84         | 6.33           | 85.88            | 83.47            | 80.52            | 55.49          | 19.92            | 87.95             | 87.42             | 86.38             |\n| **Ours**           | **54.41**     | 6.31           | **89.14**        | **86.56**        | **83.35**        | **59.63**      | 19.37        | **90.91**         | **90.33**         | **89.26**         |\n\n\n### Efficiency of the proposed approach:\n\nWe thank the reviewer for raising this valuable point. We agree that including efficiency measurements helps clarify the practicality of the proposed model.\nTo this end, we report the number of trainable parameters, inference time, and GFLOPS for each method alongside their performance.\n**As shown in Table R1-B, HopaDIFF achieves superior performance on the RHAS task while requiring fewer trainable parameters and lower computational cost compared to methods like ASQuery and RefAtomNet. These results highlight the effectiveness of HopaDIFF in balancing accuracy and efficiency for referring human action segmentation.**\n\n**Table R1-B: Efficiency measurement when we leverages BLIP-2 feature extractor on RHAS133, cross movie partition**\n| **Method**     | **Trainable Param** | **Inference Time** | **GFLOPS**  | **ACC (Test)** | **EDIT (Test)** | **F1@10 (Test)** | **F1@25 (Test)** | **F1@50 (Test)** |\n|----------------|---------------------|---------------------|-------------|-------------|-------------|-------------|-------------|-------------|\n| FACT           | 10.29M              | 1.98s            | 16.37G      | 27.89   | 0.44     | 59.05     | 58.37     | 57.13     |\n| ACTDIFF        | 7.7M                | 2.28s            | 7.14G       | 2.36    | 15.09    | 22.44     | 22.28     | 21.80     |\n| LTContent      | 6.12M               | 2.70s            | 20.02G      |52.52   | 0.37     | 49.35     | 47.24     | 42.55     |\n| ASQuery        | 47.28M              | 1.24s            | 57.22G      |20.84   | 0.08     | 33.86     | 32.86     | 30.20     |\n| RefAtomNet     | 106M                | 3.29s            | 273.53G     | 38.73   | 0.13     | 41.16     | 39.17     | 35.44     |\n| HopaDIFF       | 20M             | 2.81s        | 10.72G  |**59.63** | **19.37** | **90.91** | **90.33** | **89.26** |"}, {"review_id": "k3dd7sGJ8u", "reviewer": "Reviewer_rUnz", "summary": "The paper introduces the first work for addressing temporal action segmentation in multi-person scenarios. \nThe authors constructed a new dataset specifically for this new task. \nThey further propose a diffusion-based solution that integrates both global (holistic) and local (partial) image features, with the latter capturing the specific target subject being described. \nThe solution also leverages frequency-domain information to enhance diffusion-based action segmentation.\nExperiments on the proposed dataset show that the proposed solution outperforms baseline methods. \nThe ablation study further verifies the benefits of incorporating partial image features and leveraging frequency-domain information.", "questions": "1. When training and testing existing methods (i.e., FACT, ActDiff, ASQuery, LTContent, RefAtomNet), what are the inputs for these methods? \n    - Do they take the original images containing multiple people as input, or do they use image crops featuring only the person being described? \n    - Or I would say, what are the performances of existing methods if we first crop the target person based on textural description and then feed these image crops to these existing methods?\n2. As a follow-up to the first question, what if considering an ablated setup of the proposed solution, where this ablated setup does not leverage holistic features? Examining its performance could illustrate the benefits of capturing interactions among people through holistic features, which could further justify the importance of the proposed task.\n3. Regarding the evaluation protocol, if there are multiple ground truth action labels for the same person at the same timestamp (e.g., sitting and watching for person 1, Fig. 1), how do the authors determine whether the action segmentation is correct? Should the method output all ground truth labels, or is it sufficient to output only one of them?\n4. Why does leveraging frequency-domain information benefit action segmentation? It would be beneficial if the authors could explain the motivation for this design choice.\n5. For the ablated setting _w/o HP-xLSTM_ and _w/o BCAF_, compared to the full solution, what are their differences in the detailed framework design?", "limitations": "Yes, the authors have adequately addressed the limitations and potential negative societal impact in their submission.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**: \n- The paper addresses the important task of multi-person action segmentation, which has not been discussed in previous works.\n- The paper contributes a new dataset designed for this important task.\n- The proposed solution leverages image features of description-conditioned image crops and captures frequency domain information, which are proven to be effective.\n\n**Weakness**:\n- The paper lacks necessary detailed explanations regarding the implementation of existing methods and ablated settings. Regarding this issue, I have further explained in the question session.\n- I consider that the authors should discuss a naive solution that first crops the target subject based on the input textural reference and then solves by applying existing single-person action segmentation frameworks on these image crops.  \n\nTherefore, I vote for \"borderline reject\" for the current version. I hope the authors could address my concerns listed in the weakness and questions sessions.", "was_revised": true, "final_justification": "The rebuttal has addressed most of my concerns raised in my original comments. I am glad to raise my score towards a positive rate.", "rebuttal": "Dear Reviewer rUnz,\n\nThank you for taking the time to review our work and your valuable suggestions. Below, we provide our detailed, point-by-point responses to your comments.\n\n> **W1**-Implementation of the existing approaches.\n\nThank you for your comment. **We clarify that all the baselines in our main paper use the video with whole-image video as input.** We provide further point-to-point answer to your each question in this response.\n\n> **W2**-using cropped images of target person as input.\n\nThank you for your suggestion. **The ablation results in Table R2-A show that using video with only cropped person regions leads to a consistent drop in ACC compared with using whole-image video, with partial improvements in F1 scores, showing that embeddings learned on the whole video differs from the cropped video, however the leveraged single person baselines still deliver unsatisfactory performances.** Since the actions involves not only person physical movement (e.g., sitting) but also involves actions communicate with other persons and background objects (e.g., talking to). Using only cropped video will cause the performance drop on actions of the latter case.\n\n\n> **Q1**-When training and testing existing methods (i.e., FACT, ActDiff, ASQuery, LTContent, RefAtomNet), what are the inputs for these methods?\n\nThank you for your comment. **All the baseline approaches take the original images (with multi persons and w/o cropping) together with textual reference to a desired person as input.** We will clarify these details in our manuscript.\n\n> * 1. Do they take the original images containing multiple people as input, or do they use image crops featuring only the person being described?\n\nThank you for the comment. **All the baselines take the original images containing multiple people as input.** We will clarify this in our manuscript.\n\n> * 2. Or I would say, what are the performances of existing methods if we first crop the target person based on textural description and then feed these image crops to these existing methods?\n\nThank you for the valuable comment. We agree that incorporating these ablation studies strengthens our experimental analysis. Accordingly, we provide Table R2-A to present the results. To ensure a fully automated pipeline and avoid manual intervention, the bounding boxes used for cropping are obtained from GroundingDINO.\n\nWhen comparing the performance of existing methods that use only cropped person regions with Table 3 in our main paper, **we observe a consistent decline in the ACC metric across all baselines, along with partial improvements in F1 scores.** These findings highlight the complementary roles of both holistic and partial branches in referring human action segmentation. **Specifically, the holistic branch incorporates background and contextual information, which aids in recognizing actions involving multi-person communication and human-object interaction. In contrast, the partial branch focuses on the referred individual, enhancing recognition and segmentation of personal movement-related actions.** \n\n**Table R2-A: Ablation experiments with cropped video as input, when we use BLIP-2 feature extractor and cross-movie partition**\n| **Method** | **ACC (Val)** | **EDIT (Val)** | **F1@10 (Val)** | **F1@25 (Val)** | **F1@50 (Val)** | **ACC (Test)** | **EDIT (Test)** | **F1@10 (Test)** | **F1@25 (Test)** | **F1@50 (Test)** |\n|-------------------------|---------------|----------------|------------------|------------------|------------------|----------------|------------------|-------------------|-------------------|-------------------|\n| FACT| 16.73| 0.39| 66.07| 65.73| 64.69| 15.82| 0.30| 57.30| 55.75| 53.72|\n| ActDIFF| 3.84| 3.51| 51.38| 50.53| 47.87| 2.04           | 7.67| 44.13| 43.70| 42.42|\n| LTContent| 45.36| 0.08|60.51|57.00|50.87| 50.34          | 0.08| 49.58| 47.14| 42.43|\n| ASQuery| 32.42| 0.11| 51.18| 48.31| 43.32| 36.51| 0.12   | 44.46| 42.52| 39.09|\n| RefAtomNet| 27.73| 0.13| 46.90| 43.52|38.82| 30.45          | 0.15|38.01| 36.30| 32.72|\n| Ours w/ only partial branch                | 44.48     | 6.28       | 79.40        | 77.34        | 74.42        | 48.45      | **19.70**        | 81.99         | 81.49         | 80.48         |\n| **Ours w/ holistic and partial branches**| **54.41** | **6.31** | **89.14** | **86.56** | **83.35** | **59.63** | 19.37 | **90.11** | **90.33** | **89.26** |\n\n\n> **Q2-** A follow-up to the first question, what if considering an ablated setup of the proposed solution, where this ablated setup does not leverage holistic features? Examining its performance could illustrate the benefits of capturing interactions among people through holistic features, which could further justify the importance of the proposed task.\n\nThank you very much for this comment which helps us to explore more on HopaDIFF. We conducted the suggested ablation experiment and compare it with our approach in Table R2-B. **We observe that removing the holistic branch brings large performance degradation on the evaluation metrics, i.e., ACC, F1@10, F1@25, and F1@50, illustrating that the holistic information provided by the whole original images is important for the model to understand the human actions in the complex scenario, especially considering the actions which need the target person communicate with the environment and other persons**.\n\n\n**Table R2-B: Ablation experiments for HopaDIFF**\n| **Method**|**ACC (Val)**|**EDIT (Val)**|**F1@10 (Val)** | **F1@25 (Val)**|**F1@50 (Val)** |**ACC (Test)** | **EDIT (Test)**|**F1@10 (Test)**|**F1@25 (Test)** | **F1@50 (Test)**|\n|-------------------------|---------------|----------------|------------------|------------------|------------------|----------------|------------------|-------------------|-------------------|-------------------|\n|**Ours (w/o holistic branch)**|44.48     |6.28|79.40|77.34|74.42|48.45|**19.70**|81.99| 81.49|80.48|\n| **Ours**|**54.41**|**6.31**|**89.14**|**86.56**| **83.35** |**59.63**|19.37|**90.11**|**90.33**| **89.26** |\n\n\n\n> **Q3**-Regarding the evaluation protocol, if there are multiple ground truth action labels for the same person at the same timestamp (e.g., sitting and watching for person 1, Fig. 1), how do the authors determine whether the action segmentation is correct?\n> Should the method output all ground truth labels, or is it sufficient to output only one of them?\n\nThank you for your thoughtful question. \nIn this work, we adopt standard action segmentation metrics, including per-frame accuracy (ACC), EDIT distance, and F1 scores at temporal IoU thresholds of 10%, 25%, and 50%. Since fine-grained actions often involve multiple co-occurring labels (e.g., \"sitting\" and \"watching\"), **we adapt these metrics to a multi-label setting**.\n\nTo account for this, **we evaluate the model’s predictions on a per-frame basis by comparing the top-k predicted labels with the ground-truth annotations, where k is the number of ground-truth labels provided for that frame. If the predicted top-k set exactly matches the ground-truth labels for that frame, we treat the prediction for that frame as correct**. This true positive judgement is used in every aforementioned metric.\n\n> **Q4**-Why does leveraging frequency-domain information benefit action segmentation? It would be beneficial if the authors could explain the motivation for this design choice.\n\nThank you for your question. We are happy to elaborate on the motivation behind leveraging frequency-domain information as condition for diffusion based human action segmentation.\n\nIn our task of referring human action segmentation, accurately identifying the temporal transitions of fine-grained actions is critical. Traditional diffusion-based models rely only on temporal and spatial feature conditions to guide the generation process. However, such representations may not capture the full spectrum of temporal dependencies, especially subtle or periodic patterns that are important for fine-grained action understanding.\n\nTo address this, we incorporate frequency-domain information by applying the DFT along the temporal axis of the learned spatial and temporal embeddings. **Fourier condition allows the diffusion action segmentation model to be conditioned on global temporal characteristics, such as periodic motion patterns or transitions between actions, which are not always apparent in the raw time domain.** By conditioning the diffusion process with these frequency-based cues, the model gains finer control over the generation process, improving both the temporal precision and stability of the action segmentation outputs.\n\nOur ablation study (Table 5 in the paper) confirms the benefit of this design choice: removing frequency-domain conditioning leads to a notable drop in F1 scores across all thresholds. This validates the frequency information's role in enhancing segmentation quality, particularly in complex, multi-person scenarios.\n\n> **Q5**-For the ablated setting w/o HP-xLSTM and w/o BCAF, compared to the full solution, what are their differences in the detailed framework design?\n\nThank you for the question. In the w/o HP-xLSTM ablation, we remove the entire HP-xLSTM module, meaning that there is no more xLSTM based temporal reasoning in the framework. We directly use the extracted features from BLIP-2 as input to the ASFormer encoder and decoder to formulate the diffusion training and inference process. This limits the model’s ability to integrate global and person-specific cues, reducing temporal alignment and segmentation quality.\n\nIn the w/o BCAF ablation, we keep the HP-xLSTM structure for temporal modeling, but we disable the cross-branch attention gating at the input stage. The experimental results in Table 5 in our main paper show that both ablations result in decreased performance, especially in F1 scores, highlighting the importance of both the HP-xLSTM architecture and the BCAF mechanism in enabling strong long-range, target-aware temporal modeling."}, {"review_id": "BaCJuXu9Pj", "reviewer": "Reviewer_DuhU", "summary": "The authors introduce Referring Human Action Segmentation (RHAS), a novel task for segmenting the actions of a specific individual in complex, multi-person videos guided by a textual reference. To facilitate research in this new area, the authors present the RHAS133 dataset. The primary technical contribution is HopaDIFF, a diffusion-based framework designed to address this task. The model integrates both holistic (full-frame) and partial (person-specific) information through a dual-branch architecture. Its key innovations include a novel cross-input gate attentional xLSTM to enhance long-range temporal reasoning and a Fourier-based conditioning mechanism to exert more precise control over the generation of action segments.", "questions": "The paper presents a novel and effective framework, but further clarification and analysis would benefit a few points.\n- Could the authors provide qualitative results to offer a more complete picture of the model's performance?\n- Could the authors replace HP-xLSTM with a more standard cross-attention mechanism (similar to that used in FACT [1]) to see if the advanced memory and gating features of xLSTM [2] are truly necessary for achieving state-of-the-art results?\n\nOther details are reported in the weaknesses.\n\n[1] Lu, Zijia, and Ehsan Elhamifar. \"Fact: Frame-action cross-attention temporal modelling for efficient action segmentation.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n\n[2] Beck, Maximilian, et al. \"xLSTM: Extended Long Short-Term Memory.\" Advances in Neural Information Processing Systems 37 (2025): 107547-107603.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Paper Strengths\n- **Well-Defined Motivation**: Most existing human action segmentation methods are tailored for single-person scenarios [1,2,3,4,5], where actions occur in a fixed sequence defined by a specific task. To address the limitations of these approaches in more complex settings, the authors propose a new task called Referring Human Action Segmentation (RHAS), which focuses on multi-person untrimmed videos guided by textual descriptions.\n- **New Dataset Introduction**: The authors present a new dataset named RHAS133, which includes 133 untrimmed videos featuring a total of 542 individuals. Each person of interest is annotated with a corresponding textual reference, supporting the RHAS task.\n- **Introduction of diverse baselines**: To highlight the difficulty of the RHAS task, the authors adapt and evaluate a range of existing state-of-the-art methods for temporal action segmentation, originally designed for single-person videos, within the RHAS setting. These adapted methods serve as a baseline for comparison and demonstrate the novel challenges of the task.\n- **Introduction and Detailed Explanation of HopaDIFF**: The authors propose a novel framework named HopaDIFF (Holistic-Partial aware Fourier-conditioned Diffusion Framework). It includes two branches: a holistic branch that captures the global context from the entire video, and a partial branch that uses GroundingDINO to locate the referred individual, generating a cropped video for fine-grained representation via a Vision-Language Model (VLM). To support efficient reasoning over long videos and integrate holistic and partial information, the authors introduce the HP-xLSTM module, a modified version of the mLSTM [6] module that incorporates a bidirectional cross-attention (BCA) mechanism at the input gate. Uniquely, this framework integrates Fourier conditioning, offering finer control over the generative process by operating in the frequency domain.\n- **Effectiveness of HopaDIFF**: As shown in Section 5 of the paper, HopaDIFF significantly outperforms baseline methods, demonstrating strong performance gains. The authors conduct an ablation study, analyzing the impact of different VLM backbones and showing that while backbone quality matters, HopaDIFF consistently outperforms competitors. Additional ablations highlight the individual contributions of each component within the framework.\n\n## Paper Weaknesses\n- **Absence of Qualitative Results**: The paper's evaluation is entirely quantitative. While the reported metrics are strong, the paper would be significantly improved by including qualitative results. Visual comparisons are essential for building intuition about the model's performance on this complex task. It would be beneficial for the authors to include:\n    - Side-by-side visualizations of the segmentation output from HopaDIFF compared to key baselines on selected video segments.\n    - A dedicated analysis of failure cases to provide insight into the model's current limitations.\n- **Limited Scope of Ablation Studies**: The provided ablation study effectively demonstrates the contribution of each component within the HopaDIFF framework. However, a more compelling analysis would involve testing the modularity of these components by integrating them into the baseline architectures. This would provide stronger, more isolated evidence of their value. Specific suggestions include:\n    - *Integrating the Partial Branch into FACT [1]*: Since the FACT [1] baseline also uses a dual-branch architecture, its action branch could be replaced with the proposed GroundingDINO-based partial reasoning branch. This would directly test the effectiveness of this component outside of the full HopaDIFF model.\n    - *Augmenting ActDiff with Frequency Conditioning*: The authors note that the ActDiff [2] baseline operates on temporal and spatial features. A valuable experiment would be to augment ActDiff [2] with the proposed DFT-based frequency conditions to determine if this mechanism provides a general performance benefit to diffusion-based action segmentation models.\n    - *Comparing HP-xLSTM to Standard Attention*: The ablation study shows a performance drop when removing the HP-xLSTM module. However, to better justify its complexity, it would be insightful to replace HP-xLSTM with a more standard cross-attention mechanism (similar to that used in FACT [1]) to see if the advanced memory and gating features of xLSTM [6] are truly necessary for achieving state-of-the-art results.\n\n$\\underline{\\text{Minor Revision}}$:\nIn Figure 2, there is a typo: Patial -> Partial.\n\n[1] Lu, Zijia, and Ehsan Elhamifar. \"Fact: Frame-action cross-attention temporal modeling for efficient action segmentation.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n\n[2] Liu, Daochang, et al. \"Diffusion action segmentation.\" Proceedings of the IEEE/CVF international conference on computer vision. 2023.\n\n[3] Bahrami, Emad, Gianpiero Francesca, and Juergen Gall. \"How much temporal long-term context is needed for action segmentation?.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.\n\n[4] Gan, Ziliang, et al. \"ASQuery: A Query-based Model for Action Segmentation.\" 2024 IEEE International Conference on Multimedia and Expo (ICME). IEEE, 2024.\n\n[5] Peng, Kunyu, et al. \"Referring atomic video action recognition.\" European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2024.\n\n[6] Beck, Maximilian, et al. \"xLSTM: Extended Long Short-Term Memory.\" Advances in Neural Information Processing Systems 37 (2025): 107547-107603.", "was_revised": true, "final_justification": "I was already convinced of the potential of this paper, and the rebuttal and discussion phase further strengthened my confidence. The authors have thoroughly addressed all of my concerns. I continue to recommend this paper for acceptance.", "rebuttal": "Dear Reviewer DuhU,\n\nThank you for recognizing the contribution of our work, your comments and the review effort. We provide point-to-point responses to your concerns as follows,\n\n> **W1**-Absence of Qualitative Results: The paper's evaluation is entirely quantitative. While the reported metrics are strong, the paper would be significantly improved by including qualitative results. Visual comparisons are essential for building intuition about the model's performance on this complex task.\n>  It would be beneficial for the authors to include:\n> 1. Side-by-side visualizations of the segmentation output from HopaDIFF compared to key baselines on selected video segments.\n> 2. A dedicated analysis of failure cases to provide insight into the model's current limitations.\n> \n> **Q1**-Could the authors provide qualitative results to offer a more complete picture of the model's performance?\n\nThank you very much for your valuable suggestion. We fully agree that including more qualitative results and a detailed analysis of failure cases can provide deeper insight into the strengths and limitations of our proposed method and the newly introduced task.\n\nDuring the rebuttal phase, we prepared a set of qualitative results and failure case analyses to address this point. However, as stated in the official email from the conference, sharing images during the rebuttal is strictly prohibited. Therefore, we were unable to include these visual results in our response. But we still deliver the observations as follows.\n\nFor the qualitative results, we conducted a comparative analysis among our proposed method (HopaDIFF), ActDIFF, and FACT. We observed that HopaDIFF consistently produces action segmentation outputs that more closely align with the ground truth labels. **Notably, our method demonstrates superior performance in handling periodic action transitions along temporal axis. This improvement can be attributed to the high granularity of the condition offered by the Fourier embeddings, especially on periodic temporal patterns, and the enhanced temporal reasoning capabilities of our proposed holistic-partial aware cross input-gate xLSTM architecture towards different visual focus.**\n\nIn terms of failure case analysis, we identified a performance drop in classes with limited training samples. **In such long-tail scenarios, frames corresponding to rare classes are often misclassified as more frequent classes. We acknowledge this limitation and consider tackling the long-tail distribution challenge in referring human action segmentation as a promising and important direction for future research.**\n\nWe will incorporate these qualitative comparisons and failure analyses in the final version of our manuscript to provide a more comprehensive evaluation of our approach.\n\n> **W2**-Limited Scope of Ablation Studies:  The provided ablation study effectively demonstrates the contribution of each component within the HopaDIFF framework. However, a more compelling analysis would involve testing the modularity of these components by integrating them into the baseline architectures. This would provide stronger, more isolated evidence of their value. Specific suggestions include:\n> 1. Integrating the Partial Branch into FACT [1]: Since the FACT [1] baseline also uses a dual-branch architecture, its action branch could be replaced with the proposed GroundingDINO-based partial reasoning branch. This would directly test the effectiveness of this component outside of the full HopaDIFF model.\n> 2. Augmenting ActDiff with Frequency Conditioning: The authors note that the ActDiff [2] baseline operates on temporal and spatial features. A valuable experiment would be to augment ActDiff [2] with the proposed DFT-based frequency conditions to determine if this mechanism provides a general performance benefit to diffusion-based action segmentation models.\n> 3. Comparing HP-xLSTM to Standard Attention: The ablation study shows a performance drop when removing the HP-xLSTM module. However, to better justify its complexity, it would be insightful to replace HP-xLSTM with a more standard cross-attention mechanism (similar to that used in FACT [1]) to see if the advanced memory and gating features of xLSTM [6] are truly necessary for achieving state-of-the-art results.\n\n> **Q2-** Could the authors replace HP-xLSTM with a more standard cross-attention mechanism (similar to that used in FACT [1]) to see if the advanced memory and gating features of xLSTM [2] are truly necessary for achieving state-of-the-art results?\n\n\nThank you very much for your comment. We agree with you that the mentioned ablations are interesting and could enhance our experimental validation of the proposed approach. We thereby provide more ablations in Table R3-A.\n\n**Table R3-A: More Ablation Results**\n\n| Method                    | Val Acc | Val EDIT | Val F1@10 | Val F1@25 | Val F1@50 | Test Acc | Test EDIT | Test F1@10 | Test F1@25 | Test F1@50 |\n|---------------------------|---------|----------|-----------|-----------|-----------|----------|-----------|------------|------------|------------|\n| FACT [37]        | 26.01   | 0.34 | 60.16 | 57.79 | 55.33 | 27.89   | 0.44 | 59.05 | 58.37 | 57.13 |\n| FACT w/ partial branch         | 18.35   | 0.32     | 62.57     | 61.10     | 61.10     | 18.34    | 0.43      | 63.72      | 63.35      | 62.40      |\n| ActDiff [31]     | 4.46    | 5.96 | 38.86 | 38.36 | 37.51 | 3.26    | 15.09| 22.44 | 22.28 | 21.80 |\n| ActDIFF w/ DFT-cond         | 10.46   | 6.29     | 26.61     | 24.80     | 23.40     | 11.92    | 19.54     | 30.01      | 29.64      | 28.92      |\n| Cross-Attention Fusion           | 51.74   | **6.41**     | 86.91     | 84.47     | 81.50     | 57.02    | **19.87**     | 88.97      | 88.41      | 87.36      |\n| **Ours**         | **54.41** | 6.31 | **89.14** | **86.56** | **83.35** | **59.63** | 19.37 | **90.11** | **90.33** | **89.26** |\n\n\n\n### 1. FACT ablation with partial branch. ### \nWhen comparing FACT and FACT w/ partial branch, formulated as suggested, we observe that substituting the action branch with the partial branch leads to improvements in F1@{10,25,50}. However, this substitution also causes a decline in the ACC metric. This is because the holistic branch captures broader interaction-related actions, such as those involving communication with other individuals or background context. Replacing it with the partial branch results in the loss of important contextual information. On the other hand, the partial branch shifts the learning focus to the target individual, which enhances recognition of personal movement-related actions. Movement-related actions, such as sitting, typically span longer temporal durations. **As a result, replacing the action branch with the partial branch helps the model better capture these temporally extended patterns by concentrating on the motion of the target individual. \nThis targeted focus improves the model’s ability to accurately localize long-duration actions over time, resulting in higher temporal intersection over union and consequently better performance on F1@{10,25,50}, which accounts for temporal overlap in evaluation. However, this shift in focus comes at the cost of overlooking broader contextual cues, which are important to the actions with communication with background context, resulting in a trade-off in performance, as reflected in the ACC metric.**\nThese dynamics explain the performance variations shown in Table R3-A.\n\n### 2. ActDIFF ablation with frequency condition. ###\nWe present an ablation study combining ActDIFF with the frequency-domain condition in Table R3-A. **Compared to the original ActDIFF, this variant shows improvements across most evaluation metrics, with a notable gain in the EDIT score. This suggests that the frequency condition helps the ActDIFF model better capture action boundaries during segmentation. However, we also observe a drop in F1 scores on the validation set. Given that the validation and test sets are drawn from different movies, this decline indicates that, without the support of the HP-xLSTM module, the frequency condition alone has limited generalization capability across varying movie contexts in the referring human action segmentation task.**\n\n\n\n### 3. Compare HP-xLSTM with standard cross attention. ###\n\nIn our ablation study, we compare the variant using Cross-Attention Fusion with our HopaDIFF. The results show that cross-attention fusion is indeed effective, achieving 51.74% Acc, 6.41% EDIT, and F1 scores of 86.91% (F1@10), 84.47% (F1@25), and 81.50% (F1@50) on the validation set. On the test set, this variant reaches 57.02% Acc, 19.87% EDIT, and F1 scores of 88.97%, 88.41%, and 87.36% at the respective thresholds.\n\n**While this variant performs comparably to HopaDIFF with HP-xLSTM in terms of the EDIT score, it falls short on other key metrics, particularly Acc and F1 across all thresholds. This performance gap highlights the effectiveness of our proposed HP-xLSTM module, which offers a more structured and fine-grained mechanism for long-range temporal modeling and bidirectional information exchange between the holistic and partial branches.** Unlike cross-attention fusion, which applies attention globally between branches, HP-xLSTM introduces cross-input gated attention that enables each branch to selectively incorporate temporal controls from the other on their individual features from gating perspective. This leads to better temporal segmentation and improved per-frame accuracy, particularly in complex, multi-person scenarios with fine-grained actions.\n\n> Minor revision\n\nThe mentioned typo will be corrected in our manuscript. \n\n\nWe sincerely thank the reviewer once again for the insightful ablation suggestions, which help to enrich the ablation design and analyses of our paper. All corresponding experiments and analyses will be incorporated into the revised manuscript."}, {"review_id": "UMdl799jlX", "reviewer": "Reviewer_dRDv", "summary": "This paper pioneers a novel task—Referring Human Action Segmentation (RHAS)—which extends traditional single-person temporal action segmentation to multi-person, textual reference-guided scenarios that are inherently more unconstrained and unstructured. The authors introduce RHAS133, the first dataset for this task, comprising 133 untrimmed movies with 542 annotated individuals. Each person is described with textual references and labeled with fine-grained action annotations. Furthermore, the authors propose HopaDIFF, a holistic-partial aware Fourier-conditioned diffusion framework that integrates dual-branch modeling, HP-xLSTM with bidirectional cross-attention (BCA), diffusion-based generation, and frequency-domain conditioning. The proposed approach significantly outperforms both conventional temporal action segmentation and referring action recognition baselines.", "questions": "1.The end of the Introduction section would benefit from a clear summary of the main contributions of this work.\n\n2.Regarding the dataset, there is a significant class imbalance and long-tailed distribution: some action categories contain over 1e+6 frames, while others have fewer than 1e+3. Although such imbalance is common in many datasets, the authors should explain the extent of this imbalance in detail, discuss its impact on model performance, and describe any mitigation strategies adopted—or justify the choice not to address it. This is critical for ensuring transparency and the credibility of the results.\n\n3.The Methodology section lacks sufficient architectural details for several modules. Specific concerns include:\nA. The model appears to follow a two-stage pipeline—first extracting Holistic and Partial features, then performing subsequent modeling—but this is not clearly stated or described in the paper.\nB. For the denoising diffusion model, the internal structure of the denoising function $M_{\\theta}$ is not detailed. The paper does not explain how the model processes the noisy input, timestep, and the spatial, temporal, and frequency embedding conditions during denoising.\nC. The Bidirectional Cross-Attention (BCA) module lacks a thorough explanation of its mechanism and role within the architecture.\n\n4.Concerns of the Innovation:\nA. While the use of existing components (e.g., VLM and Grounding DINO for feature extraction) is technically reasonable, key modules in the proposed framework—such as the diffusion-based segmentation and xLSTM—are borrowed directly from prior work. The paper allocates significant space and equations to these existing methods, while offering limited discussion of the truly novel aspects such as dual-branch interaction and frequency-conditioned denoising, which weakens the perceived originality.\nB. The explanations of the novel components remain superficial. For instance: 1) The rationale for using frequency-based conditioning is not sufficiently elaborated; the current explanation (\"primarily leverage features in the temporal and spatial domains...\") does not justify why frequency information is beneficial for denoising. 2) The mechanism of BCA-based information exchange is not clearly explained.  3) The decision to keep the two branches separate (rather than fuse them) and train them independently is not justified, nor is this design explored through ablation studies.\n\n5.Efficiency Issues: The paper lacks a discussion of efficiency in terms of parameter count, FLOPs, and training/inference time. The use of two VLM backbones, one Grounding DINO, two ASFormer encoders and decoders, and a diffusion model—each computationally heavy—raises concerns about the feasibility and efficiency of the two-stage pipeline.\n\n6.Performance and Fairness Issues: In Table 5, the performance of the model without HCMGB is still lower than that of FACT, despite both relying solely on holistic features. This raises questions about the comparative effectiveness of other modules in HopaDIFF, and an explanation is needed.\n\n7.Experimental Issues: \nA.Why do the test set results exceed those on the validation set in most experiments?\nB. The overall EDIT scores are low; a justification is needed.\nC. Table 5 lacks an ablation study for removing the diffusion model entirely.\nD. There are no component-level ablation studies examining the internal structure or parameters of each individual module.\nE. There is no analysis of training each branch independently or of fusing the branches—such ablations are essential for validating the dual-branch design.\n\n8.Errors and Inconsistencies:\nLine 104: \"state of the art\" should be hyphenated as \"state-of-the-art\".\nLines 85–88: The sentence beginning with “since...” and ending with “which...” is overly long and should be split for clarity.\nLine 283: The reported F1@50 score is inconsistent with the corresponding table.\nTable 5 and Lines 310 & 315 refer to “w/o DCT-cond,” which suggests Discrete Cosine Transform, while the earlier sections describe frequency conditioning based on DFT (Discrete Fourier Transform); this inconsistency needs to be clarified.\n\nThis work presents a task and framework with acceptable levels of innovation and effort. However, there remain multiple critical concerns regarding methodology, clarity, originality, and experimental completeness. I assign a Borderline Accept and will adjust my score based on the authors’ responses and revisions.", "limitations": "The paper discusses dataset limitations and future directions in both the conclusion and supplementary materials, and provides a fairly comprehensive analysis of the societal impact, including both positive and negative aspects. However, it lacks a clear discussion of the limitations of the proposed framework itself.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 4, "originality": 2, "strengths_and_weaknesses": "Quality: This work initiates the RHAS task, proposes the RHAS133 dataset, and develops the HopaDIFF framework. The technical methodology is sound, and experiments demonstrate state-of-the-art performance in this emerging area. However, the paper lacks sufficient ablation studies and does not provide an in-depth analysis of each innovative component.\nClarity: The motivation and overall structure are clearly presented. However, critical architectural details are missing, limiting reproducibility. For instance, the interaction mechanism between the two branches within HP-XLSTM via BCA, the internal structure and information flow of the denoising model at each diffusion step, and how spatial, temporal, and frequency embedding conditions guide the model are not sufficiently explained.\nSignificance: This work pioneers the RHAS task, expanding action segmentation into a more complex, textual-reference-guided multi-person scenario, and introduces a corresponding dataset and framework. The contribution is of high significance to the field.\nOriginality: The paper introduces a novel research task and a new dataset. While the proposed framework includes several innovative modules, many components (e.g., diffusion model, xLSTM) are adapted or combined from existing works. Thus, while the task-level originality is high, the architectural novelty is relatively limited.", "was_revised": true, "final_justification": "Within the limited space, the author has provided thorough and detailed responses that sufficiently address the majority of my concerns. These revisions have significantly improved the manuscript's quality, clarity, and overall rigor. Therefore, I recommend accepting this paper.", "rebuttal": "Dear Reviewer dRDv,\n\nThank you for the review effort and valuable comments. We provide point-to-point responses as follows,\n\n> **Q1**-Introduction summary.\n\nWe will add the following summary into the end of the introduction section.\n\n**To summarize, this paper makes three key contributions: \n(1) we introduce, for the first time, the task of Referring Human Action Segmentation (RHAS), paving the way for action segmentation in complex, multi-person scenarios guided by natural language; (2) we present RHAS133, the first dataset for this task, and establish a comprehensive benchmark using diverse evaluation settings and adapted baseline methods; (3) we propose HopaDIFF, a diffusion-based framework tailored for RHAS, featuring the novel HP-xLSTM module for long-term temporal modeling through cross-input gate attention between holistic and partial branches together with additional fourier condition. Our method achieves state-of-the-art performance on the RHAS133 benchmark.**\n\n> **Q2**-Long-tailed distribution problem.\n\nThank you for your comment. **Our dataset is not collected according to a fixed protocol and is based on in-the-wild data, which results in the unbalanced and long-tailed issue. In-the-wild data typically exhibits a long-tailed distribution because real-world activities naturally vary in frequency, common actions (e.g., walking, talking) occur far more often than rare ones (e.g., fainting, waving a flag) [1].** This imbalance poses a challenge for the referring human action segmentation pipelines, which tend to favor frequent classes while underperforming on rare but often critical ones. Despite this, in-the-wild data is crucial for developing robust and generalizable models, as it captures the diverse, unstructured, and noisy nature of real-world environments [2]. **In order to mitigate the negative effects we use class weight with inverse relationship to the frequency of each class to weight the loss function.** Training and evaluating on in-the-wild data improves a model's practical applicability in unconstrained settings for human action segmentation, such as surveillance, healthcare, or human-robot interaction. We will modify the manuscript accordingly to deliver the above discussion.\n\nReferences:\n\n[1] Yang, Lu, et al. \"A survey on long-tailed visual recognition.\" IJCV 2022.\n\n[2] Sigurdsson et al., \"Hollywood in Homes: Crowdsourcing Data Collection for Activity Understanding,\" ECCV 2016.\n\n> **Q3**-The Methodology section lacks sufficient architectural details, A..., B..., C....\n\nA: We will clarify that **our method adopts a two-stage approach to extract holistic and partial features in the main paper.**\n\n\nB: We use ASFormer encoders and decoders according to ActDIFF to build the diffusion network, **where noisy input, timestep embeddings, and spatial, temporal, and frequency embeddings are first encoded using their own projection network (consisting of 2 linear layers) and then summed together.** Before the projection, timestep is first encoded with a sinusoidal positional embedding.\n\n\nC: **The BCA module enables information exchange between holistic and partial branches by using cross input-gate attention to modulate input information for each stream jointly, supporting richer temporal reasoning. HP-xLSTM leverages cross input-gate attention (i.e., BCA applied to input gates), enabling each branch to selectively incorporate temporal cues from the other based on feature-level gating.** This design enhances temporal segmentation and boosts per-frame accuracy, particularly in challenging multi-person scenarios with fine-grained actions.\n\n\nThe attention weights and fused representations are computed as follows:\n\n$\\alpha_t^{H \\rightarrow P} = \\text{softmax}\\left(\\frac{(W_Q X^H_t)(W_K X^P_t)^\\top}{\\sqrt{d}}\\right)$\n\n$\\tilde{X}^P_t = \\alpha_t^{H \\rightarrow P} \\cdot (W_V X^H_t)$\n\n$\\alpha_t^{P \\rightarrow H} = \\text{softmax}\\left(\\frac{(W_Q^* X^P_t)(W_K^* X^H_t)^\\top}{\\sqrt{d}}\\right)$\n\n$\\tilde{X}^H_t = \\alpha_t^{P \\rightarrow H} \\cdot (W_V^* X^P_t)$\n\nWhere $W_Q, W_K, W_V$ and  $W_Q^* , W_K^* , W_V^* $ are projections for queries, keys, and values for the two branches. $\\tilde{X}^P_t$ and $\\tilde{X}^H_t$ are partial and holistic inputs of HP-xLSTM. All the illustration in this response will be added into our manuscript.\n\n> **Q4**-Concerns of Innovation.\n\nWe will enrich the discussion of dual-branch interaction and frequency-conditioned denoising. **Our novelty lies on the proposed holistic-partial aware xLSTM for temporal reasoning, where we propose bi-directional cross input-gate attention to enable holistic and partial aligned input control, and frequency condition, which is used to preserve temporal coherence and fine-grained details on motion dynamics by guiding the model to maintain consistency across both low and high-frequency components. Well-learned Fourier embeddings can better simulate temporal motion dynamics of specific frequencies.** The details of BCA information exchange are introduced in the response of Q3. Training two branches separately leads to better performance compared with the counterpart (two branches are fused together after HP-xLSTM) as in Table R4-A.\n\n**Table R4-A**\n|Method|Val ACC|Val EDIT|Val F1@10|Val F1@25|Val F1@50| Test ACC|Test EDIT|Test F1@10|Test F1@25|Test F1@50|\n|---------------------------|---------|----------|-----------|-----------|-----------|----------|-----------|------------|------------|------------|\n|Fused|52.12|**6.42**|87.10|84.51|81.49|56.50|**19.89**| 88.28| 87.69|86.64|\n|**Ours**|**54.41**|6.31|**89.14**|**86.56**| **83.35**|**59.63**|19.37|**90.11**|**90.33**| **89.26**|\n\n> **Q5**-Efficiency Issues.\n\nWe agree that efficiency is crucial for practical use, and thus report trainable parameters, inference time, and GFLOPs alongside performance. **As shown in Table R4-A, HopaDIFF achieves state-of-the-art results on RHAS while being more efficient than methods like ASQuery and RefAtomNet, highlighting its superiority on RHAS tasks.**\n\n**Regarding the two-stage pipeline, we note that most existing methods (e.g., FACT, ActDIFF, LTContext, ASQuery) also adopt this approach due to the high computational cost of long video processing. We acknowledge this as an open challenge in the action segmentation field and will discuss it in the limitations and future work section.**\n\n**Table R4-B**\n|**Method**|**Trainable Param**|**Inference Time**|**GFLOPS**|**ACC (Test)**|**EDIT (Test)**|**F1@10 (Test)**|**F1@25 (Test)**|**F1@50 (Test)**|\n|----------------|---------------------|---------------------|-------------|-------------|-------------|-------------|-------------|-------------|\n|FACT|67.44M|1.98s|16.37G|27.89|0.44|59.05|58.37|57.13|\n|ActDIFF|7.7M|2.28s|7.14G|2.36|15.09|22.44|22.28|21.80|\n|LTContent|6.12M|2.70s|20.02G|52.52|0.37|49.35|47.24|42.55|\n|ASQuery|47.28M|1.24s|57.22G|20.84|0.08|33.86|32.86|30.20|\n|RefAtomNet|106M|3.29s|273.53G|38.73|0.13|41.16|39.17|35.44|\n|HopaDIFF|20M|2.81s|10.72G|**59.63**|**19.37**|**90.91**|**90.33**|**89.26**|\n\n> **Q6**-Performance and Fairness Issues:\n \nWe recognize this as a limitation of existing diffusion action segmentation method. **As shown in Table 3 of the main paper, where a cross-movie partition is used, consistent with the setup in the module ablation, ActDIFF underperforms FACT. This suggests that the diffusion action segmentation model with only holistic features struggles to generalize across different movies.** Incorporating an additional partial branch, as in our approach, proves beneficial for enhancing the generalizability of diffusion-based action segmentation across diverse video styles. We will modify our paper accordingly.\n\n\n> **Q7**-Experimental Issues.\n\nA: For the cross-movie partition, **movies are randomly assigned to validation and test sets, preserving distribution shifts delivered by different movie styles.** In contrast, **under the random partition (Table 2 &4 in main paper), the performance gap between test and val sets is relatively small.**\n\n\nB: Since **our task focuses on fine-grained action annotations, recognizing precise action boundaries is inherently difficult,** leading to lower EDIT scores.\n\n\nC: As shown in Table R4-C, **removing the diffusion module causes a significant performance drop on EDIT score, highlighting its importance for action boundary reasoning.**\n\n\nD: **Module ablation results are provided in Table 5 in our main paper. The ASFormer encoder-decoder has 10.6M parameters, while HP-xLSTM adds around 9.4M.**\n\n\nE: Table R4-D presents the performance of each branch, demonstrating that **combining both holistic and partial branches yields the best overall performance.**\n\n**Table R4-C**\n|Method|Val ACC|Val EDIT|Val F1@10|Val F1@25|Val F1@50|Test ACC|Test EDIT|Test F1@10|Test F1@25|Test F1@50|\n|---------------------------|---------|----------|-----------|-----------|-----------|----------|-----------|------------|------------|------------|\n|w/o Diffusion|53.43|0.09|67.35|63.68|81.50|57.47| 58.56|0.10|56.17|53.72|48.92|\n|**Ours**|**54.41**|**6.31**|**89.14**|**86.56**| **83.35**|**59.63**|**19.37**|**90.11**|**90.33**| **89.26**|\n\n**Table R4-D**\n|Method|Val ACC|Val EDIT|Val F1@10|Val F1@25|Val F1@50|Test ACC|Test EDIT|Test F1@10|Test F1@25|Test F1@50|\n|---------------------------|---------|----------|-----------|-----------|-----------|----------|-----------|------------|------------|------------|\n|w/o partial branch|20.78|**6.34**|45.81|43.93|41.95|23.44|**19.93**| 50.60|50.14|49.42|\n|w/o holistic branch|44.48|6.28|79.40| 77.34|74.42|48.45|19.70|81.99|81.49|80.48|\n|**Ours**|**54.41**|6.31|**89.14**|**86.56**|**83.35** |**59.63**|19.37|**90.11**|**90.33**|**89.26**|\n\n> **Q8**-Errors and Inconsistencies.\n\nThank you for pointing out the writing issues, we truly appreciate your effort. We will carefully correct the typos and inconsistent descriptions in the manuscript. Regarding the Fourier condition, yes, it should be “w/o DFT-cond,” as we use the DFT in our method."}]

Author rebuttal:

Meta-review: This paper proposes a new dataset, RHAS133, for textual reference-guided human action segmentation in multi-person settings. Based on the dataset, a new method is designed and evaluated. The method is based on a diffusion model for action segmentation, incorporating multiple modifications such as cross-input attention and Fourier conditions.

Reviewers commented positively regarding the novel task proposed, the new dataset contributed, the reasonable method design, and the comprehensive experiments. Reviewers commented negatively mainly on the limited technical novelty, combining existing components, missing ablation studies of complicated components, and the absence of qualitative results. The weaknesses have been well addressed after the rebuttal.

The main reason for acceptance is the new task and the new dataset, advancing the field of action segmentation. Action segmentation has been saturated for a relatively long time, and the classic formulation does not keep up with vision-language foundation models. This paper, with its new task and dataset, will provide new opportunities for the community.

The dataset, evaluations, and codes should be fully released as a benchmark to facilitate future research.
Decision: Accept (spotlight)