ActFusion: a Unified Diffusion Model for Action Segmentation and Anticipation
Authors: Dayoung Gong
Suha Kwak
Minsu Cho
Venue: NeurIPS 2024 poster (2024)
Keywords: temporal action segmentation
long-term action anticipation
Primary area: machine_vision
TL;DR: a unified diffusion model for temporal action segmentation and long-term action anticipation

Abstract: Temporal action segmentation and long-term action anticipation are two popular vision tasks for the temporal analysis of actions in videos. 
Despite apparent relevance and potential complementarity, these two problems have been investigated as separate and distinct tasks. In this work, we tackle these two problems, action segmentation, and action anticipation, jointly using a unified diffusion model dubbed ActFusion. 
The key idea to unification is to train the model to effectively handle both visible and invisible parts of the sequence in an integrated manner;
the visible part is for temporal segmentation, and the invisible part is for future anticipation. 
To this end, we introduce a new anticipative masking strategy during training in which a late part of the video frames is masked as invisible, and learnable tokens replace these frames to learn to predict the invisible future.
Experimental results demonstrate the bi-directional benefits between action segmentation and anticipation.
ActFusion achieves the state-of-the-art performance across the standard benchmarks of 50 Salads, Breakfast, and GTEA, outperforming task-specific models in both of the two tasks with a single unified model through joint learning.

ActFusion: a Unified Diffusion Model for Action Segmentation and Anticipation
Dayoung Gong
Suha Kwak Minsu Cho
Pohang University of Science and Technology (POSTECH) {dayoung. gong, suha. macho)0postech. ac.kr
Abstract
Temporal action segmentation and long-term action anticipation two popular vision tasks for the temporal analysis of actions in videos. Despite apparent rele- vance and potential complementarity, these two problems have been investigated as separate and distinct tasks. In this work, we tackle these two problems, ac- tion segmentation and action anticipation, jointly using a unified diffusion model dubbed ActFusion. The key idea to unification is to train the model to effectively handle both visible and invisible parts of the sequence in an integrated manner, the visible part is for temporal segmentation, and the invisible part is for future anticipation. To this end, we introduce a new anticipative masking strategy during training in which a late part of the video frames is masked as invisible, and learnable tokens replace these frames to learn to predict the invisible future. Experimental results demonstrate the bi-directional benefits between action segmentation and anticipation. ActFusion achieves the state-of-the-art performance across the stan- dard benchmarks of 50 Salads, Breakfast, and GTEA, outperforming task-specific models in both of the two tasks with a single unified model through joint learning.
1 Introduction
In everyday life, when interacting with people, we anticipate their future actions while recognizing their actions observed in the present and the past. Similarly, for effective human-robot interaction, robotic agents must recognize ongoing actions while anticipating future behaviors. Two essential tasks in computer vision for such temporal understanding of human actions are temporal action segmentation (TAS) [38, 19, 32, 61, 12, 65, 3, 43, 63, 8] and long-term term action anticipation (LTA) [2, 34, 25, 45]. The task of TAS aims at translating observed video frames into sequence of action segments, while the goal of LTA is to predict a plausible sequence of actions in the future based on the observed video frames. These tasks are closely related in terms of understanding the relations between actions; recognizing actions in the present and the past may improve anticipating action in the future, and the ability anticipate the future may also enhance recognizing observable actions when facing visual ambiguities.
Despite the apparent relevance and potential complementarity. these two problems have been investi- gated as separate and distinct tasks. While a growing body of work has shown remarkable progress on both TAS and LTA, these methods are primarily designed for one task (see Fig. la) and show poor generalization when applied to the other (e.g., see FUTR [25] and DiffAct [43] in Fig. Ic). Some prior methods tackle both tasks simultaneously, but they rely task-specific architectures and require separate training for each individual task (e.g., see TempAgg [52] in Fig. Ic).
In this paper, introduce ActFusion, unified diffussonn model that addresses TAS and LTA through a single architecture and training process, as illustrated in Fig. 1b. The core idea of our unification lies in training the model to effectively handle two different parts of the sequence: visible parts for action segmentation and invisible parts for action anticipation. Accordingly, we propose a
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

TAS
LTA
Seg.
Ant.
Seg.
Seg.
Ant.
75.1
75.5
54.3
TAS model
LTA model
ActFusion
12.6
observation
abservatin
anticiation
observatoo
disersaiion antoppation
FUTR
TempAgg
(a) task-specific models
(b) ActFusion (ours)
(c) evaluation on TAS LTA
Figure I: Task-specitic models vs. ActFusion (ours). (a) Conventional task-specific models for TAS and LTA. (b) Our unified model ActFusion address both tasks. (c) Performance comparison across tasks. Tasks-specific models such as DiffAct [43] for TAS and FUTR [25] for LTA exhibits poor performance on cross-task evaluations. ActFusion outperforms task-specific models on both TAS and LTA, including TempAgg [52], which trains separate models for each task. Note that the performance of ActFusion is evaluation result of a single model through a single training process. The reported performance represents the average of each task in the original paper or evaluated with the official checkpoint (See Sec. G for details).
new anticipative masking strategy in which late part of the video frames is masked invisible, and learnable tokens replace it. We also introduce a random masking strategy to help the model accurately identify actions when some parts of the video ambiguous [59], drawing inspiration from previous methods [15, 6, 28, 59]. ActFusion generates action sequences from Gaussian noise through iterative denoising, conditioned process, the model jointly learns to classify action labels of visual features and predict future actions of mask tokens. As shown in Fig. Ic, ActFusion achieves superior performance in both action segmentation and anticipation, demonstrating stronger cross-task generalization compared to previous methods [43, 52].
both visual features and mask tokens. Through this
Furthermore, we find that the benchmark evaluation of some prior LTA methods [2, 20, 25] has exploited the ground-truth length of input videos in testing time, which makes the evaluation unrealis- tic and problematic since the duration of future actions is supposed to be unknown advance. To address this issue, we conduct comprehensive experiments both with and without ground-truth length information, providing insights into more realistic deployment scenarios.
Our contribution can be summarized as follows: 1) We introduce ActFusion, a unified diffusion model that jointly addresses TAS and LTA through single training process. 2) We present anticipative masking for effective task unification, along with random masking to enhance robustness against visual ambiguities. 3) Comprehensive experimental results show that jointly learning both tasks provides bi-directional benefits. ActFusion achieves state-of-thee -ft---rtttt-tttt performance on TAS and LTA across benchmark datasets 50 Salads and Breakfast, and GTEA demonstrating the effectiveness of the proposed method.
Related work
Temporal action segmentation (TAS). The goal of TAS [19, 65, 43, 63, 12, 23, 5, 8, 32, 3, 16] is to classify frame-wise action labels in video. Earlier approaches employ temporal sliding windows [50,33 33] for action segment detection, grammar-based methods [37, 36) have been intro- duced to incorporate temporal hierarchy of actions during segmentation. Temporal convolution networks [38, 19] and Transformer-basse models [65, 8] are introduced based on deep learning methods. Since learning long-term relations of actions from activity videos is challenging, series of work has been proposed to develop refinement strategies [30, 32, 23, 12, 57, 63, 3, 24] that can be applied to the TAS models [19, 65). Recently, DiffAct [43] is proposed to iteratively denoising action predictions conditioned on the input video features adopting the diffusion process. a similar spirit, the proposed ActFusion is based on the diffusion process, focusing on unifying TAS and LTA through anticipative masking. We demonstrate the bidirectional benefits between the two tasks, showing that learning segmentation along with anticipation is effective.
Long-term action anticipation (I.TA). LTA [2, 1, 34, 20, 52, 25, 44, 26] has recently emerged as a crucial task for predicting sequence of future actions in long-term videos. Initial models use RNNs

---

and CNNs [2]. while conditondt anticipation [34] introduces one-shot anticipation of specific future timestamps. A GRU-based model [20] used cycle-consister relations of past and future actions. Sener et al. [52] proposed TempAgg, a multi-granular temporal aggregation method for action anticipation and recognition, utilizing different model architectures and task-speeiiic losses for the two tasks. Gong et al. [25] propose a transformer model for parallel anticipation, dubbed FUTR, empirically showing that learning action segmentation as an auxiliary task is helpful for anticipation. Nawhal et al. [44] propose a two-stage learning approach for LTA and Zhang et al. [66] present object-centric representations using Visual-language pre-trained models for LTA. While previous work [52, 20, 25, 44] adopts TAS as auxiliary task to help learn LTA, unified model evaluated on both tasks is rarely explored, often showing poor cross cross-task generalization performance (Fig. Ic). The exception is TempAgg. which requires separate training and model architecture for the two tasks. In this work, we present a unified model that tackles both TAS and LTA in a single training process.
Diffusion models. Recent success in denoising diffusion models [55, 29, 56] opens a new era of computer vision research. The diffusion models learn the original data distribution through the iterative denoising process. Diffusion models have recently shown notable success in various domains, such as image generation [51, 14], video generation [42, 27, 64], object detection [13], semantic segmentation [7], temporal action segmentation [43], self-supervised learning [62], and etc [18]. Recently, DifIMAE [62] integrates masked autoencoders with the diffusion models, where the model learns to denoise masked input while learning data distributions through generative pre-training of visual representations. In this work, we present a unified diffusion model that effectively integrates TAS and LTA through masking, where the model learns denoise action labels conditioned on both visual and mask tokens. In this way, the model can effectively learn temporal relations between actions by classifying visual tokens and inferring missing actions of the mask tokens.
3 Preliminary
In this section, we give a brief overview of diffusion models [29, 54]. The diffusion models learn data distribution by mapping and denoising noises from the original data distribution. The training process of diffusion models involves forward and reverse processes from random noise. During the forward process, Gaussian noise is added to the original data, while in the reverse process, a neural network learns reconstruct the original data by iteratively removing noise.
The forward process, or diffusion process, transforms the original data x. into noisy data x":
x V>(s)> + VI -ss)c
(1)
Here, noise ((0, I) is added to the original data distribution x. following the decreasing function 7s8) of time step s E {1, ... where S represents the entire forward time-step. Note that the function >ss) determines the intensity of the noise added to the original data following the pre-defined variance schedule [29].
In the reverse process, a neural network f(z",)) is trained to recover the original data x from noisy data x" using l2 regression loss:
L C= -=111 =012
1
During inference, the model iteratively denoises pure noise x to reconstruct the original data x i.e., I - S + + x following the updating rule [29, 54]. We refer the reader to [29, 54] and Sec. A for more details.
In our context, the neural network learns to generate action sequences from Gaussian noise, condi- tioned on visible features for action segmentation and mask tokens for action anticipation.
4 Proposed approach
We present ActFusion, unified diffusion model for action segmentation and anticipation. This section describes the problem setup in See. 4.1, the model architectrre in Sec. 4.2, and the proposed masking strategies and training objectives in Sec. 4.3 and Sec. 4.4, respectively.
3

---

Masking
Conditioning
[M]
[M]
EM]
[M]
[M]
enc
dec /
dec
MM
[M]
h
[M]
[M]
M]
[M]
T
none
ant
randon boundary ret
F
E
A²
7
1 leraivv dencising A
Training:
Inference: none (TAS), ant (LTA)
masks
visual tokens
[M mask tokens
Training
Inference
Figure 2: Overall pipeline of Actousion, During training, we randomly select one of five masking strategies and apply it to input video frames F, replacing masked regions with learnable tokens to obtain masked features F". These features are processed by the encoder to produce visual embeddings E, which condition the decoder l to denoise action labels from A to A at time-step For inference, we use different masking strategies depending on the task: no masking for TAS and anticipative masking for LTA. The decoder then iteratively denoises action labels following A using the DDIM update rule [54].
As A
4.1 Problem setup
Temporal action segmentation (TAS) aims to classify input video frames into a sequence of predefined action classes, while long-term action anticipation (L.TA) predicts future actions based on partially observed video sequences. Formally, given video sequence F [ii, F2, ................................ Fr] of length T, TAS predicts frame-wise action labels A [A1, A2, ...... Ar), where each A, is a one-hot vector representing the action class. In LTA, given the first No [al'l observed frames, the goal is to anticipate action labels for the subsequent A [3T]] frames, where a E [0, 1] and / E [0. 1 al represent the observation and anticipation ratios, respectively. Here, [-] denotes the ceiling function.
4.2 ActFusion
ActFusion aims to unify TAS and LTA by leveraging an encoder-decoder architecture with adaptive masking strategies. Figure illustrates the overall pipeline, where ActFusion consists of a masked encoder yg and a denoising decoder h. The encoder obtains visual features and mask tokens input and generates embedded tokens as output. The decoder then progressively reduces noises through an iterative denoising process conditioned on the embedded tokens.
During training, we randomly sample a time step E {1, 21.,...., S} for each iteration and add noise <~ N (0, I) to the ground-truth action labels A following Eq. 1, resulting in noisy action labels A'. The decoder then aims to denoise A to reconstruct the original action labels A". During inference, the decoder starts with Gaussian noise Á and progressively denoises following the DDIM [54] update rule to generate final predictions 1 i.e., A Á + Åo
The key to our unified approach lies training the model to effectively process both visible and invisible parts of the sequence, where the visible part corresponds to observed video frames and the invisible part represents future frames to be anticipated. To this end, we introduce anticipative masking that replaces unobserved frames with learnable mask tokens, enabling the model to learn future predictions. We apply this masking strategy consistently during both training and inference to achieve joint learning of action segmentation and anticipation. We further incorporate random masking, where video frames are randomly masked to enhance robustness against visual ambiguities. Both masking strategies are detailed in Sec. 4.3.
Input structuring. Given input video features F E RTxC with T frames of feature dimensions, we start by defining binary mask matrix M {0, 1 This mask serves as frame selector, where a value of indicates frame be masked and replaced by mask tokens, while a value of indicates an unmasked frame. The learnable mask token, denoted as I glxc' replaces the visual

---

features in frames selected for masking, producing the input features F" for the encoder:
=' F M (1TxC M) o m,
(3)
where @ denotes element-wise multiplication and 1 represents a matrix of ones with dimensions i j. Here, M and m are broadcasted along the channel and temporal dimensions, respectively.
For our model architecture, we use a modified version of ASFormer [65] used in DiffAct [43], which employs dilated attention to capture both local and global relations in the input sequence (See Fig. S2 for the detailed model architecture).
Encoder. The encoder consists of the N layers, each consisting of dilated 1-d convolution followed by instance normalization, dilated attention, and a feed-forward network [65]. In the dilated attention mechanism, the receptive field is limited to local window size w = 2 for the i-th layer, where the increasing dilation captures progressively broader temporal relations. The output of each layer is combined with its input via a residual connection before proceeding to the subsequent layer. Given the input features F", the encoder 9 produces embedded tokens E E T as:
E = F(F),
(4)
where D represents the dimensions of embedded tokens. fully-connected (FC) layer W R K is applied to E to obtain frame-wise classification logits from the encoder followed by softmax:
A euc = o EW
(5)
where K is the number of action classes and is the softmax.
Decoder. The decoder is composed of ND sequential layers. Each layer consists of dilated 1-d convolution, dilated attention, instance normalization, and feed-forward networks. As in the encoder, the dilation ratio for the i-th layer is set to w = 2'. The output of each layer is combined with its input through residual connection before being passed into the subsequent layer. Given a time step noisy action labels A at time step and encoder embeddings E, the decoder produces output embeddings Y RTxD according to:
Y h(A", s, E).
(6)
The final action label predictions A E R are obtained by projecting Y through fully-connected
layer dree E IR Dx followed by softmax:
A = = oY
4.3 Masking strategy
We introduce two distinct masking strategies:
anticipative masking and random masking.
Anticipative masking. Anticipative masking enables joint learning of TAS and LTA by separating visible and invisible parts of the input sequence. This strategy masks the latter portion of video frames, requiring the model to predict future actions based on observed frames. Given video length T, we define a binary anticipation mask M E 1) 1)" that sets visible frames to I and invisible frames to
O: M = (i < NO). where is an indicator function and No represents the number of observed frames. Unlike causal masking [60] that prevents attending to future tokens, our anticipative masking allows interactions among visible tokens while maintaining clear boundary for anticipation.
Random masking. Random masking aims provide robustness in prediction when parts of video frames are missing or ambiguous [59]. A video is first divided into pre-defined clips of size Q, resulting in NP -6= total clips. Then, R clips are randomly selected to be masked. A binary random mask is defined by M. 1 E P,, - Q)Q < i < jQ), where IP is a randomly selected subset of {1. (1....,N'' " with [P NR.
For fully observable scenarios, we utilize a no mask strategy M where all frames remain visible. Additionally, we adopt two masking strategies [43], the relation mask M and the boundary mask M as illustrated in Fig. 2. The relation mask M randomly masks segments associated with an action class to learn inter-action dependencies. The boundary mask R masks frames at action transitions to enhance boundary detection. See Sec. C for the detailed formulations of the masks.
During training, one of five masking strategies is randomly selected: mask M anticipative mask M random mask M relation mask M and boundary mask M For inference, no mask M" is used for TAS and anticipative mask A for LTA.
5

---

Table 1: Comparison with state of the art TAS. The overall results demonstrate the efficacy of ActFusion on TAS, achieving otate-of-the-art performance across benchmark datasets. Bold values
represent the highest aceuracy, while underlined values indicate the second- highest accuracy.
50 Salads [58]
Breakfast [36]
GTEA [2]]]
methods
F1
Avg.
F1
50)
Acc.
Avg-
FL 25, 50
Acc.
Avg.
MS-TCN [19]
76.3/ 61.5
67.9
80.7
72.7
52.6/
61.7
66.3
53.3
83/4/69.
79.0
76.3
78.9
MS-TCN++ [4]]
80,7 78.5 70.1
74.3
83.7
77.5
64.1/58.6/4
/S.6//59
65.6
67.6
60.4
85.8 /6.77766.0
K3.5
80.1
82.8
SSTDA [17]
83.0781.55 73.8
75.8
83.2
79.5
75.0 69.1 /55.2
73.7
70.2
68.6
90.07 88.0/89.11
19.1/ 79.0
86.2
79.8
84.6
GTRM [61]
754/7288/00 63.9
82.6
72.4
54.0/43.5
58.7
65.0
55.7
79.3
70.4
87.17
84.4
79.8
MIDA [[]]
82.0780. 72.5
75.1
83.2
78.5
74.2/ 68.6 56.5
73.6
71.0
68.8
90.5
88.4 76..
85.8
80.0
84.2
Global2iocal [23]
80.3 78.0 69.8
73.4
82.2
76.7
74.9 69.0//9.0/550 55.2
73.3
70.7
68.6
$9.9
75.8
B1.6
78.5
83.2
HASR [3]
86.6/85.77 78.5
83,9
83.1
74.7 69.5 57.0
71.9
69.4
68.5
/
88.6 76.4
87.5
78.7
84.4
72.4
77.3
ASFormer [65]
83.1783.41 76.0
79.6
85.6
81,9
76.0/ 70.67 57.4
75.0
73.5
70.5
90.1
88.8 79.2
B1.5
79.7
81.5
ASFurmer- + KARI [24]
83.4783.8777 77.4
79.9
85.3
82.4
78.8 73.7 60.8
77.8
74.0
73.0
-7-1-
Temporal Agr: [57]
-f-f-
59.2 53.9 39.5
54.5
64.5
$4.3
-----
UARI (10)
85.3/83.577
81.8
/9.47477
67.8
79.6
86.9
DPRN [46]
84.5
7
70.1
92.0
88.1
SEDT [35]
89.97 $8.7 81.1
84.7
86.5
86.2
-1-1-
93.7
7 84.0
91.3
81.3
88.5
TCT: [+]
87.5 56.1 60.2
833.1
$6.5
84.8
76.6/7L1/ 7.5.5
76.1
77.5
72.0
91.3 90.1 88.00
87.9
81.1
86.1
FAMMSDTN [17]
86.2/84.172 77.9
79.9
56,4
83.0
78.5
75.9/
77.5
74.8
72.8
91.6 90.9 80.9
88.3
80.7
86.5
87.1/85.7/1
73.9
UVAST [8]
89.1787.6/ 81.7
83.9
87.4
85.9
76.9 7
77.1
69.7
70.6
92.7 91.3/8..
92.1
80.2
87.5
BePrompi [40]
89.2/87.8/1 81.3
83.8
88.1
86.0
-1-1-
94.1 /
91.6
81.2
88.4
MCFM [31]
90.6 89.5 84.2
84.6
90.3
87.8
-1-1-
91.8 9118/9912 80.8
88.00
80.5
86.5
L.CComtet:
87.7
96.0
T16/
7700
14.2
72.3
DiffAct (43)
35.0
81.4
64.6
78.4
26.4
75.1
91.5/
89.6
82.2
88.1
ActFasion (ours)
91.67 90.7 84.8
36.0
99.3
88.5
(76.) 64.7
79.3
26.4
75.5
94 93.3 86.9
91.6
81.9
89.6
4.4
Training objective
The model is trained with three types of losses: cross-entropy loss _ for frame-wise classification, boundary smoothing loss _smo [19]. and boundary alignment loss Lbd [43]. These losses are applied to both encoder and decoder, where the encoder serves as an auxiliary task to enhance discrimination ability. Given the ground-trute action label A E RTT K and the predictions E RT** K the cross entropy loss is defined by:
K A log A.,.
(A),
T
1=
To prevent over-segmentation errors, a temporal smooth loss [19] between adjacent frames based on truncated mean squared error over the frame-wise log probabilities are defined by:
T-1 (log A.,; log Ait1s)?
Lsmo (A)
Σ Σ
- 1
where the difference of the log probabilities of two adjacent frames is truncated with threshold value. For precise boundary detection, the boundary alignment loss [43] employed based on the binary cross-entropy loss:
Loa BB A)
T Σ (-
B, log(1 [
A. A.AA))) (1 (1) log Art1)),
(10)
where = x(B) represents a softened version of the ground-trus action boundary B, = 144 + 1)),, achieved through a Gaussian kernel The encoder and decoder losses are defined by
-ene_ = xene Leac Ceac(A" (A + xenc Cone (Aeuc) +
Abd Coac (B, Áenc) eec
(11)
dd
dec edee (A A der) Ader Liter (Adee) + dec Edre BB A dorr) X
(12)
where À denotes a scaling factor. The total loss is defined as Liotal Leue + dec
5 Experiments
In this section, we conduct experiments to demonstrate the effectiveness of our model. All reported experimental results are obtained from inference using single unified model. We evaluate

---

Table 2: Comparison with the state of the art LTA. The overall results demonstrate the effectiveness of ActFusion, achieving new SOTA performance in LTA. Bold values represent the highest accuracy, while underlined values indicate the second-highes accuracy.
3 [ax 0.2)
[ax 0.3)
dataset
input type
methods
0.1
0.2
0.3
0.5
0.1
0.2
0.3
0.5
RNN [2]
30.06
25.43
18.74
13.49
30.77
17.19
14.79
09.77
CNN
21.24
19.03
15.98
09.87
29.14
20. 14
17.46
10.86
label
UAAA (mode) [1]
24.86
22.37
19.88
12.82
29.10
20.50
15.28
12.31
Time-Cond. [34]
32.51
27.61
21.26
15.99
35.12
27.05
22.05
15.59
50 Salads
Temporal Agg- [52]
25.50
19.90
18.20
15.10
30.60
22.50
19.10
11.20
Cycle Cons. [20]
34,76
28.41
21.82
15.25
34.39
23.70
18.95
15.89
A-ACT [26]
35,40
29.60
22.50
16.10
35.20
25.30
20.10
16.30
feature
FUTR [25]
39.55
27.54
23.31
17.77
35.15
24.86
24.22
15.26
ObjectPrompt [66]
37.40
28.90
24.20
18.10
28.00
24.00
24.30
19.30
ActFusion (ours)
39.55
28.60
23.61
19,90
42.80
27.11
23.48
22.07
RNN [2]
18.11
17.20
15.94
15.81
21.64
20.02
19.73
19.21
CNN [2]
17.90
16.35
15.37
22.44
20.12
19.69
18.76
label
UAAA (mode) [1]
16.71
15.40
14.47
14.20
20.73
18.27
18.42
16.86
Time-Cond. [34]
18.41
17.21
16.42
15.84
22.75
20.44
19.64
19.75
Breukfast
Temporal Agg. [52]
24.20
21.10
20.00
18.10
30.40
26.30
23.80
21,20
Cycle Cons. [20]
25.88
23.42
22.42
21.54
29.66
27.37
25.58
25,20
feature
A-ACT [26]
26.70
24.30
23.20
21.70
30.80
28.30
26.10
25,80
FUTR [25]
27.70
24.55
22.83
22.04
32.27
29.88
27.49
25.87
ActFFusion (ours)
28.25
25.52
24.66
23.25
35.79
31.76
29.64
28.78
method on three widely-used benchmark datasets: 50 Salads [58]. Breakfast [36], and GTEA [21]
(see Sec. F for details). All three datasets are used to evaluate on TAS, while 50 Salads and Breakfast are used for evaluating LTA,, following the protocols of the previous work [19, 65, 43, 20, 52, 25).
Evaluation metrics. For evaluation metrics for TAS, we report F1 @ (10, 25, 50] scores, the edit score, and frame-wise accuracy [19, 65, 43]. The Fl scores and the edit score are segment-wise metrics, and accuracy is frame-wise metric. Mean over classes accuracy (MoC) is adopted as an evaluation metric for LTA [2, 34, 52, 25].
Implementation details We utilize the pre-trained I3D features [9] as input video features for all datasets provided by [19]. For the diffusion process, set the entire time step S 1000 [29, 54], with a skipped time step for inference set to 25 [54]. For the anticipation mask M we set the observation ratio E {0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8). For the random mask M fix the size of patch w to 10 for both tasks and randomly select the number masked patches N to 25, 10, and 20 for 50 Salads, Breakfast, and GTEA, respectively. For the encoder and decoder, adopt modified version of ASFormer [65] using relative positional bias where the maximum number of neighbors is set to 100. During inference, we a and to for TAS [65, 19, 43]. For LTA, we set a E {0.2, 0.3} and B E (0.1, 0.2, 0.3, 0.5), following the evaluation protocols [2, 20, 52, 25]. See Sec. G for more details.
5.1 Comparison with the state of the art on TAS and LTA
Tables 1 and 2 present performance comparisons across benchmark datasets, where our single unified model achieves superior results on both tasks. Table shows the results on TAS, where ActFusion outperforms all task-specific models demonstrating the benefits of joint learning. Table compares LTA performance across different datasets and input types: predicted action labels of the visual features [49] and pre-trained I3D features [9]. Overall, ActFusion achieves state-of wrate-of-the-an -he-tarrtt performance on two benchmark datasets, demonstrating the efficacy of joint learning for TAS and LTA based on the diffusion process. Note that we do not include the performance of ANTICIPATR [44] due to differences evaluation setup, as reported in [67].
5.2 Analysis
We conduct comprehensive analyses to validate the effectiveness of the proposed method. In the following experiments, we evaluate our approach on the 50 Salads dataset. All experimental settings the same explained in Sec. unless otherwise specified.
7

---

Table 3: Segmentation helps anticipation
8 (m 0.2)
(a 0.3)
C
0000
0.1
0.2
0.3
0.1
0.2
0.3
27.32
21.97
18.04
17.47
17.78
14.82
13.23
14.24
34.51
26.26
19.69
15.84
28.75
20.33
20.03
16.95
39.55
28.60
23.61
42.80
23.48
22.07
Table 4: Ablation studies masking
(a) Results on TAS
(b) Results on LTA
M M M M
F 10, 25, 50}
Edit
Aoc.
Avg.
M M M
ar" M
0.2)
(I)
91.6 90.7 84.8
86.0
89.3
88.5
0.1
0.2
0.3
0.5
90.0 88.9 83,9
84.7
88.6
87.2
(1)
42.80
27.11
23.48
22.07
(3) V V
89.9 89.0 82.4
83.9
88.7
86.8
89.9 89.3 83.9
83.7
88.3
87.0
(3)
38.60
25.57
20.42
18.62
(5)
V V 90.2
89.2 83.7
85.0
87.7
87.2
(4)
V
37.23
25.29
20.82
20.35
Segmentation helps anticipation. To validate the impact of learning TAS LTA, conduct loss ablation experiments by removing the action segmentation loss on the observed No frames. Table 3 shows the results. Specifically,
exclude the encoder loss Le in Eq. 11, while omitting the decoder loss _dee in Eq. 12 on the observed frames. For simplicity, we denote the decoder loss for the observed frames and unobserved frames to be anticipated Lgee and L respectively. Note that Egee utilized for effective anticipation learning. By comparing (1) and (3) in Table 3, observe a significant performance drop when removing the loss on the observed frames. We observe that applying segmentation loss the observed frames in the decoder improves performance when comparing (1) and (3). The overall results show that learning action segmentation plays a crucial role in improving action anticipation.
Anticipation helps segmentation. To validate the effect of jointly learning anticipation along with segmentation, conduct ablation studies anticipative masking M in Table 4. Comparing (I) and (2) Table 4a, we find that using anticipation masking helps improve the performance of TAS. In comparison the results in Table 4, where segmentation greatly enhances anticipation, the performance improvement without using anticipation masking is slightly lower. Nevertheless, we find that anticipation does contribute positively to the overall metrics of segmentation.
Ablation studies on masking types. To evaluate the effects of different types of masking on TAS and LTA, we conduct ablation studies in Table 4. The performance TAS and LTA are presented in Table 4a and 4b, respectively. We observe that anticipative masking M plays a significant role in the joint learning of TAS and LTA, evidenced the substantial performance drop in LTA when comparing row (2) with the other rows in Table 4b. Random masking M significantly reduces performance in both TAS and LTA, as shown in row (3) of Table 4. Furthermore, the use of boundary masking M and relation masking M is also essential for both tasks. Due to the limited space, we only report the performance LTA when the observation ratio is set to 0.3.
Effects of learnable mask tokens. To verify the effects of learnable mask tokens m, we replace m with zero vectors. Table 5 shows the overall results. Comparing rows (1) and (2) in Table 5, we find that using learnable mask tokens more effective for both TAS and LTA. We conjecture that this is due to the mask tokens being embedded with the visual tokens within the encoder, which aids in effectively anticipating the invisible parts.
Position of mask tokens. Instead of providing mask tokens as input to the encoder, we conduct experiments on providing mask tokens input to the decoder, shown in row (3) in Table 5. In this experiment, only the visible frames are given as input to the encoder, while mask tokens m' E R x are applied in the decoder to fill the original masked positions, similar to the approach used in masked auto-encoder [28]. By comparing rows (2) and (3) in Table 5, we observe that using mask tokens the encoder more effective than using them the decoder. We hypothesize that embedding mask tokens alongside visual tokens within the encoder benefits joint learning of TAS and LTA. Furthermore, when mask tokens are provided only to the decoder, they not receive the encoder loss gene, which may negatively impact performance.

---

Table 5: Analysis mask tokens
(a) Results on TAS
(b) Results on LTA
Pos.
Lear.
F1 (10, 25, 50)
Edit
Acc.
Avg.
Pos
Leum.
(ix
(n
90.5 89.1 83.3
84.3
87.4
86.9
0.1
0.2
0.3
055
0.1
0.2
0.3
0.5
91.6 90.7 84,8
86.0
89.3
88.5
6x
39:55
35.4
28.80
27.51
2361
21.88
26.70
19.00
38.13
27:18
26.91
13.48
22.69
22317
21.88
doc
89.9 88.9 82.1
84,2
88.1
86.7
dec
23.61
18.00
12.13
1730
TTS
L=LTA
-TA
-TTAA
Avg. porooma
87.8
88.5
Avg pee wwweoommneee
Ang par peffrmmeen
87.9
88.3
28.4
28.5
66.9
86.6
27.2
26.9
86.0
26,5
100
The number of masked dips X
The number masked clipa A
The number of inference steps
The number of inference stops
Figure 3: The number of masked clips NR Figure 4: Inference steps of diffusion process
The number of masked clips NR in random masking. To determine the number of masked clips NR in random masking, we conduct experiments by adjusting N from 10 to 100 while fixing the window size Q of each masked clip N to 10. Figure 3 illustrates the result, suggesting that employing appropriate amount of masking is crucial.
The number of inference steps in the diffusion process. We compare performance according to the number of inference steps of the diffusion process in Fig. 4. We observe consistent performance increases as the number of inference steps increases. As the increasing step requires more computation, we choose to use 25 steps for inference in all of our experiments.
5.3 Qualitative results
Figure 5 presents qualitative results evaluated both TAS and LTA using a single model. Each figure includes video frames, ground-truth action sequences, and predicted results for TAS and LTA. Only the visible parts (observed frames) used input during inference LTA. Overall results show that ActFusion effectively handles both visible and future segments, accurately classifying current actions and anticipating future ones. Additional results are provided in Fig. S3 and S4.
5.4 Evaluation without using ground truth prediction length on LTA
Benchmark evaluations in previous work, including those of [20, 25] and our results presented above, have been typically conducted following the evaluation protocol of [2] where prediction length N is set to 3T in testing; 3 and T are the prediction ratio and the ground-truth video length, respectively. This can be seen a leakage of ground-trui information in testing because the exact length of future actions is supposed to be inherently unknown during inference in real-world scenarios. In this subsection, thus rectify the evaluation setting by determining the prediction length rNO based solely on the number of observed frames NO, where r is a hyperparameter that adjusts the relative length of future predictions. We then train method with this modified anticipation masking strategy, denoted as ActFusion In this experiment, r is set to 4.
Using the rectified evaluation protocol, we compare ours with Cycle Cons. [20] and FUTR [25], whose codes are available For a fair comparison, we modify both models to use the same prediction length r denoted Cycle Cons. and FUTRÍ. The results summarized in Table 6 where the numbers in parentheses indicate relative performance changes compared to those in Table 2. Table 6 shows that all the methods exhibit overall performance degradation when ground-truth length information not used. These observations reveal that previous results have benefited from the use of ground-trun --utttuthhhhhu length T. leading to information leakage in testing. Therefore, suggest that future research avoid relying on this information for more realistic and fair comparison. On the other hand, the results also show that our approach consistently outperforms the methods across benchmark datasets, demonstrating its effectiveness even without ground-truth prediction length information. Please refer Sec. G for further details of the experimental setup.
'The official code for Cycle Cons. [20] was obtained directly from the authors, and the official code repository for FUTR [25] is available at ht https: //gi thub. ctm/ gongdaOe/FUTi,

---

activity: fried eeg
TAS
LTA
observation
buster pan
egg
saltmpepper
iyy egz
put epglplate
(a) Breakfast
activity: making salad
GT
observation
anticipation
action start
cut tonato
place tomato into bowl
chock:
place checse into bowl
cut lettuce
place bout
cucumber oucumber
place cucumber
ingredients adi vinegae
pepper
mix dressng
serve salad onto plate
add dressing action end
(b) 50 Salads
Figure 5: Qualitative results
Table 6: Evaluation without using ground truth prediction length on LTA
datuset
methhods
(a - 0.2)
/ (cx 0.3)
0.1
0.2
0.3
0.5
D.1
0.2
0.3
0..
Cycle Cons. [20)
31.85 (2.91.)
28.19
23.98 (2.10)
(Q.77)
(7.511
(8.36(5.541)
(4.52
(5.54
50 Salads
FUTR
20.00 (2.46)
(.800.
ActP usion (ours)"
35.96.3.69.)
28.09 (0.1))
24.20 (0:591
20.13
41.14 (1.660)
25.68 (1.43)
23.13 23.12
18.35
Cycle Coas. [20]
25.66 (0.23.)
23.10 (0.32))
21.66 (0.76))
19.73 (1.82))
29.08 (0.59)
25.68 (1.71)
23.00 (2.991)
20.54 (4.36.)
Breakfast
FUTR
27.85 (0.10")
24.68 (0.15t)
22.91 (U99))
20.98
32.36 (0091)
28.96
25.69
23.56 (2.13L)
(ours)"
(0.70.)
24.69 (0.32))
22.99 (2.581)
22.42(0.84))
33.47 (1.75))
30.28 (1.22L)
30,69 (1.121)
26.77 (2.02.)
6 Conclusion
We have presented ActFusion, a unified diffusion model that jointly addresses temporal action segmentation and long-term action anticipation in videos. The key our method is anticipative masking, where learnable mask tokens replace unobserved frames, enabling simultaneous action segmentation of visible parts and anticipation of invisible parts. Our comprehensive experiments demonstrate that this unified approach not only outperforms existing task-specific models on both tasks but also reveals the mutual benefits joint learning. Additionally, by evaluating our method both with and without ground-truth length information during LTA inference, we hope to motivate future research toward not using this information during testing. We believe that ActFusion demonstrates the potential of unifying complementary tasks in temporal action understanding, opening new directions for future research.
7
Acknowledgement
This work was supported by Samsung Electronics (I0201208-07822-01) and IIT grants (RS-2022- I1220959: Few-Shot Learning of Causal Inference in Vision and Language for Decision Making (50%), RSS-2022-12202244: Comprehensive Video Understanding and Generation with Knowledge- based Deep Logic Neural Network (45%), RS-2019-1119999
AI Graduate School Program at POSTECH (5%) funded by Ministry Science and ICT, Korea. We thank reviewers for pointing out the issue of the evaluation protocol and providing insightful discussions.
10

---

References
[1] Y. Abu Farha and J. Gall. Uncertainty- anticipation of activitiess In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops, pages 0-0, 2019.
[2] Y. Abu Farha, A. Richard, and J. Gall. When will you do whatt-anticipating temporal occur- rences of activities. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages
2018. 2,2,,,7,5
[3] H. Ahn and D. Lee. Refining action segmentation with hierarchical video representations. In Proc. IEEE International Conference on Computer Vision (ICCV), pages 16302-16310, 2021. 2021.
1, 2,
[4] N. Aziere and S. Todorovic. Multistage temporal convolution transformer for action segmenta- tion. Image and Vision Computing, 128: 104567, 2022.
[5] E. Bahrami, G. Francesca, and Gall. How much temporal long-term context is needed for action segmentation? In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10351-10361, 2023. 6
[6] H. Bao, L. Dong, S. Piao, and F. Wei. Beit: Bert pre-training of image transformers. arXiv preprint arXiv:2106.0822 2021. 2
[7] D. Baranchuk, A. Voynov, Rubachev, V. Khrulkov, and A. Babenko. Label-efficient semantic segmentation with diffusion models. In International Conference on Learning Representations,
2021. 3
[8] N. Behrmann, S. A. Golestaneh, 7. Kolter, J. Gall, and M. Noroozi. Unified fully and times- tamp supervised temporal action segmentation via sequence to sequence translation. In Proc. European Conference on Computer Vision (ECCV), pages 52-68. Springer, 2022. 1, 2, 6,
[9] J. Carreira and A. Zisserman. Quo vadis, action recognition? new model and the kinetics dataset. Proc. IEEE Conference on Computer Vision and Patter Recognition (CVPR), pages 6299-6308, 2017.
[10] L. Chen, M. Li, Y. Duan, J. Zhou, and J. Lu. Uncertainty-awan representation learning for action segmentation. IJCAI, volume page 2022. 6
[11] M.-H. Chen, B. Li, Y. Bao, and G. AlRegib. Action segmentation with mixed temporal domain adaptation. In Proc. IEEE Winter Conference on Applications of Computer Vision (WACV), pages 005, 2020. 6
[12] M.-H. Chen, B. Li, Bao, G. AIRegib, and Kira. Action segmentation with joint self- supervised temporal domain adaptation. In Proc. IEEE Conference on Computer Vision and Puttern Recognition (CVPR), pages 9454-9463, 2020. 1, 1,2.6
[13] S. Chen, P. Sun, Y. Song, and P. Luo. Diffusiondet: Diffusion model for object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 19830- 19843, 2023. 3
[14] T. Chen, R. ZHANG, and G. Hinton. Analog bits: Generating discrete data using diffusion models with self-conditioning. In The Eleventh International Conference on Learning Repre- sentations, 2022. 3
[15] Devlin, M.-W. Chang, K. Lee, and Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding, In NAACL-HLT (1), 2019. 2, 18, 19
[16] G. Ding. F. Sener, and A. Yao. Temporal action segmentation: An analysis of modern techniques.
IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023. 2
[17] Z. Du and Q. Wang. Dilated transformer with feature aggregation module for action segmenta- tion. Neural Processing Letters, pages 1-17, 2022. 6

---

[18] W.-C. Fan, Y.-C. Chen, D. Chen, Y. Cheng, L. Yuan, and Y.-.. F. Wang. Frido: Feature pyramid diffusion for complex scene image synthesis. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 579-587, 3
[19] Y. A. Farha and J. Gall. Ms-ten: Multi-stage temporal convolutional network for action segmentation. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3575-3584 2019. 2, 6, 7, 17, 19
[20] Y. A. Farha, Q. Ke, B. Schiele, and J. Gall. Long-Term Anticipation of Activities with Cycle Consistency. In Proc. German Conference on Pattern Recognition (GCPR). Springer, 2020. 2, 3. 7,9, 10, 17, 20
[21] A. Fathi, X. Ren, and M. Rehg. Learning to recognize objects in egocentric activities. In CVPR 2011, pages
IEEE, 2011. 6, 7, 20
[22] C. Feichtenhofer, Y. Li, K. He, et al. Masked autoencoders as spatiotemporal learners. Advances in neural information processing systems, 35:35946-3355 35958, 2022. 19, 21
[23] S.-H. Gao, Q Han, Z.-Y. Li, P. Peng, L. Wang, and M.-M. Cheng. GlobalZlocal: Efficient structure search for video action segmentation. In Proc. IEEE Conference Computer Vision and Pattern Recognition (CVPR), pages 16805-16814 2021. 2,6 6
[24] D. Gong, J. Lee, D. Jung, S. Kwak, and M. Cho. Activity grammars for temporal action segmentation. Advances in Neural Information Processing Systems. 36, 2024. 2, 6
[25] D. Gong, J. Lee, M. Kim, S. Ha, and M. Cho. Future transformer for long-tern action anticipation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Putlern Recognition, pages 1062-3061, 2022. 1, 2, 3, 7, 9, 10, 17, 20, 21
[26] A. Gupta, J. Liu, L. Bo, A. K. Goy-Chowdhury, and T. Mei. A-act: Action anticipation through cycle transformations. arXiv preprint arXiv:2204.00942 2022. 2,7
[27] W. Harvey, S. Naderiparizi, V. Masrani, C. Weilbach, and F. Wood. Flexible diffusion modeling of long videos. Advances in Neural Information Processing Systems, 35:27953-27965, 2022.
[28] K. He, X. Chen, S. Xie, Y. Li, P. Dollár, and R. Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages 16000-16009, 2022. 2, 8, 19,
[29] J. Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models. In Proceedings of the 34th International Conference on Neural Information Processing Systems, pages 18,0-6851, 2020. 3, 7, 15
[30] Y. Huang, Y. Sugano, and Y. Sato. Improving action segmentation via graph-based temporal reasoning. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 44024-14034 2020. 2,
[31] K. Ishihara, G. Nakano, and T. Inoshita. Mcfm: Mutual cross fusion module for intermediate fusion-based action segmentation. In 2022 IEEE International Conference on Image Processing (ICIP), pages 1701-1705. IEEE, 2022.
[32] Y. Ishikawa, S. Kasai, Y. Aoki, and H. Kataoka. Alleviating over-segmentation errors by detecting action boundaries. In Proc. IEEE Winter Conference on Applications of Computer Vision (WACV), pages 2322-1222 22321, 2021. 1,2,
[33] S. Karaman, L. Seidenari, and A. Del Bimbo. Fast saliency based pooling of fisher encoded dense trajectories. In ECCV THUMOS Workshop, volume 1, page 5, 2014.
[34] Q Ke, M. Fritz, and B. Schiele. Time-conditioned action anticipation in one shot. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 9925-9934, 2019. 1,
[35] G.-h. Kim and E. Kim. Stacked encoder-decoder transformer with boundary smoothing for action segmentation. Electronics Letters, 58(25)9972-974, 2022. 6
12

---

[36] H. Kuehne, A. Arslan, and T. Serre. The language of actions: Recovering the syntax and semantics of goal-directed human activities. In Proc. IEEE Conference on Computer Vision and Puttern Recognition (CVPR), pages 780-787, 2014, 2, 6, 7,
[37] H. Kuehne, A. Richard, and J. Gall. Weakly supervised learning of actions from transcripts. Computer Vision and Image Understanding, 163:78-89, 2017. 2,
[38] C. Lea, M. D. Flynn, R. Vidal, A. Reiter, and G. D. Hager. Temporal convolutional networks for action segmentation and detection. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 156-165, 2017. 1,2
[39] J. Li, P. Lei, and S. Todorovic. Weakly supervised energy-based learning for action segmentation. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), October 2019. 21
[40] M. Li, L. Chen, Y. Duan, Z. Hu, J. Feng. J. Zhou, and J. Lu. Bridge-prompt: Towards ordinal action understanding instructional videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19880-19889, 2022.
[41] S.-J. Li, Y. AbuFarha, Y. Liu, M.-M. Cheng, and J. Gall. Ms-ten++: Multi-stage temporal convolutional network for action segmentation. IEEE transactions on pattern analysis and machine intelligence, 2020. 6
[42] X. Li, W. Chu, Y. Wu, W. Yuan, F. Liu, Q. Zhang, F. Li, H. Feng, E. Ding, and J. Wang. Videogen: reference-guided latent diffusion approach for high definition text-to-video generation. arXiv preprint arXiv:2309. 00398, 2023. 3
[43] D. Liu, Li, A.-D. Dinh, T. Jiang, M. Shah, and C. Xu. Diffusion action segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10139- 10149, 2023. 2, 5, 6, 7, 17, 19, 20.21 21
[44] M. Nawhal, A. A. Jyothi, and G. Mori. Rethinking learning approaches for long-term action anticipation. In European Conference on Computer Vision, pages 558 376. Springer, 2022. 2,
3.7
[45] M. Nawhal and G. Mori. Activity graph transformer for temporal action localization. arXiv preprint arXiv:2101.000 20010084,, 2021.
[46] J. Park, D. Kim, S. Huh, and S. Jo. Maximization and restoration: Action segmentation through dilation passing and temporal reconstruction. Pattern Recognition, 129:108766 2022.
[47] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neurul information procexsing systems, 32, 2019. 21
[48] C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Zhou, W. Li, and P. J. Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of machine learning research, 21(140):1-60 2020. 17, 18
[49] A. Richard, H. Kuehne, and J. Gall. Weakly supervised action learning with rnn based fine- to-coarse modeling. In Proc. IEEE Conference Computer Vision and Pattern Recognition (CVPR), pages 754-763, 2017. 21
[50] M. Rohrbach, S. Amin, M. Andriluka, and B. Schiele. A database for fine grained activity detection of cooking activities. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1194-1201. IEEE, 2012. 2
[51] R. Rombach, A. Blattmann, D. Lorenz. P. Esser, and B. Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages (0684-10695 2022. 3
[52] F. Sener, D. Singhania, and A. Yao. Temporal aggregate representations for long-range video understanding. Proc. European Conference on Computer Vision (ECCV), pages 154-171. Springer, 2020. 1, 2, 3, 6, 7, 17, 20
13

---

[53] P. Shaw, J. Uszkoreit, and A. Vaswani. Self-attention with relative position representations. arXiv preprint 1803.0280 2018. 18
[54] Song, C. Meng, and S. Ermon. Denoising diffusion implicit models. International Conference Learning Representations, 2020. 3, 4, 15
[55] Y. Song and S. Ermon. Generative modeling by estimating gradients of the data distribution. Advances in Neural Information Processing Systems, 32, 2019.
[56] Y. Song, Soht-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, and B. Poole. Score-based generative modeling through stochastic differential equations. International Conference on Learning Representations, 2020.
[57] Y. Souri, Y. A. Farha, F. Despinoy, G. Francesca, and J. Gall. Fifa: Fast inference approximation for action segmentation. In Pattern Recognition: 43rd DAGM German Conference, DAGM GCPR 102 Bonn, Germany, September 28-October 1, 2021, Proceedings, pages 282-296. Springer, 2022.
[58] S. Stein and S. McKenna. Combining embedded accelerometers with computer vision for recognizing food preparation activities. In Proceedings of the 2013 ACM international joint conference on Pervasive and ubiquitous computing, pages 729-738, 2013. 6, 20
[59] Z. Tong, Y. Song, J. Wang, and L. Wang. Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training. Advances in neural information processing systems, 35: 15:10008-10093 2022. 2,5
[60] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. Proc. Neural Information Processing Systems (NeurIPS), 30, 2017. 5
[61] Z. Wang. Z. Gao, L. Wang, Z. Li, and G. Wu. Boundary-aware cascade networks for temporal action segmentation. In Proc. European Conference on Computer Vision (ECCV), pages 34-51. Springer, 2020. 1,
[62] C. Wei, K. Mangalam, P.-Y. Huang, Y. Li, H. Fan, H. Xu, H. Wang, C. Xie, A. Yuille, and C. Feichtenhofer. Diffusion models as masked autoencoders. arXiv preprint arXivv2300.03223, 2023.
[63] Z. Xu, Y. S. Rawat, Y. Wong, M. Kankanhalli, and M. Shah. Don't pour cereal into coffee: Dif- ferentiable temporal logic for temporal action segmentation. In Advances in Neural Information Processing Systems (NeurIPS). 2,
[64] R. Yang, P. Srivastava, and S. Mandt. Diffusion probabilistic modeling for video generation. Entropy, 25(10):14655 2023. 3
[65] F. Yi, H. Wen, and T. Jiang. Asformer: Transformer for action segmentation. In Proc. British Machine Vision Conference (BMVC), 2021. 2, 5, 6, 7, 17,
[66] C. Zhang, C. Fu, S. Wang, N. Agarwal, K. Lee, C. Choi, and C. Sun. Object-centric video representation for long-term action anticipation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 6751-6761, 2024. 3,7
[67] Z. Zhong, M. Martin, M. Voit, J. Gall, and J. Beyerer. A survey on deep learning techniques for action anticipation. arXiv preprint arXiv:2309.1720 17257, 2023.
14

---

Appendix
In this appendix, offer detailed descriptions and additional results, which are omitted in the main paper due to the lack of space. We provide detailed explanation of diffusion models in Sec. A, training and inference algorithms in Sec. D, additional experimental results in Sec. E, more information of datasets in Sec. F, more experimental details Sec. G, more qualitative results Sec. H, and discussions of limitations and broader impacts in Sec. I.
Sec. B, masking types in Sec. C, encoder and decoder layers
A Diffusion models
In this section, we give an in-depth explanation the diffusion models [29, 54] described in Sec. 3. The forward process adds noise to data distribution x" ~ ((xx), following the noise distribution based on the Markov property. Specifically, the forward process is defined by adding noise at an arbitrary time step with variance (s)) follows:
((x' x (z0) = II -<<")x²-))
(13)
"( 1 N(x:; vI VI- (1)-7())==+_--0 (1=)0))) (s))),
(14)
where S represents the entire time step. We can directly obtain the noisy data distribution x at time step without iteratively applying q due the Markov property:
g(x (x") N(x:: ((5)2² (1 (1-8(s))) 8()))),
(15)
x² ((())= + +V (1-((s))
(16)
where ((s) : 7s)) 8(8)== IT 5(1), and NO. Instead of using ((s), 1 8(s) is utilized [29]. The posterior q(=-~1122 x²) conforms to Gaussian distribution and is expressed in terms of mean m (z", xx and variance (s) using Bayes Theorem:
a( xx),00) = N ~~==~11110 "aa"";m(a" m(x", ,,,, ())
(17)
where
m²(x", x0 )==)=== vō(s - 1)((s)
x0 vo(s ö(s 1))
(18)
1 d-)(s)
8(s)
1- - (8))
(8) 1 0(s)
(19)
With sufficiently large S and an appropriate variance schedule 7(8), the noisy data 5: follows an isotropic Gaussian distribution. Consequently, we know the reverse distribution (x*). we can sample 5 N(0, I) and reverse the process to obtain a sample from g(=²). However, since q(x*~1[x*) relies on the entire data distribution, we employ a neural network to approximate q(x*=1 (x*) (x=) as follows:
po(x=~112>>)
N(x 1 s), Eo (x s));
(20)
where my and Eg represent the predicted mean and co-variance, respectively, derived from the neural network. Predicting e or x in Eq. 16 found to be effective [29]. this work, we choose to predict r using the neural network.
15

---

Algorithm ActFusion Training
def train(f, a_gt):
f: video features [B, T,
Fround-truth action labels [B, T,
# T: number of franee
feature dinension
K: nunber of action classes
nasking input features
mask_types ['none'', 'ant', 'random', 'boundary', 'rel?]
nask_type
choice types) f.prime mask(f, mask_typo)
video enbeddings and predictions from encoder a_hat_enc encoder (T.pine)
aignal scaling
a.gt (a_gt
scale
corrupt a_gt. randint(O, normal (mean=0, sté-1) A noise: [B, T, X]
stop
s_crpt sqrt(éella_cuuprooee) a.gt egrt delta_cumprod(e) epe
predictions from decoder a_hat_dec pecoder(a_crpp) 6,
training Laseleesaaaaaammaaa (a_atta_aa___aaaa hat_ anc, a_gt) leas_enc cal_anc_ La__________e_ loas_dec (t__a___ooaa_ahattt_____aoaaaaa_ a_gt)
loss_total loss_enc loss_dec
loss_total
B Algorithms
We provide training algorithms of ActFusion in Alg. 1 and inference algorithms for TAS and LTA in Alg. and Alg. 3, respectively.
Algorithm ActFusion TAS Inference
Algorithm ActFusion L.A Inference
inference(E, stops, S):
inference steps,
[B, Т,
alpha+T,
steps: number inference steps S: number of time steps
steps: nunber of inference steps 8: nunber of tine
masking
masking
f_prima mask(f, 'none?)
f f_prine mask(f, fant!)
Encode video features ancader(t (f (f_priine)
Eacode video features ancader (f_prime)
sample noisy action label nornal (mean=0, std=1)
# eample noisy action label (nean-0, std=1)
uniforn sampling step size revarsed (1inppccc)
# uniform sampling size
staps))
timas revarsed (11napcc(1 8, steps))
[(S-1, 8-2), (8-2, 8-3), .... (1,0), (0,-1)3
[(S-1, 8-2), (S-2, s-3), .... (1,0), (0,-1)]
time_pairs list 1itt(eip(timmstt-]] (itt ri:ms(l:)))) for t_nov, t_pext in zip(tine_pairrs: predict a.0 from a.s bat decoder(a_s, e, t_now)
time_pairs list (xip(tieens for t_next in zip(tino_paire):
# predict a_0 from a.s a_hat decoder(a_s, t_now)
estinate a.s at t_next ddin_step(a_e, a_hat, t_next)
# estinate a.s at t_next ddin_stop(aas, a_hat, t_next)
return bab
return a_hat
dolta_cumprod((); cunulative product of 6,., IT:.
16

---

C Masking types
A
A genc
W
Y
KNN
add
M
add
FFN
FFN
M
add & IN
add & IN
MR
M
conscat
M
dilated conv
dilated conv
N none
A ant
R random
F'
A
S relation
M boundary
relative position bias
addition
mullipicctiion
take cup
pour coffee
pour milk
stir coffee
(b) encoder and decoder layer
Figure S1: Masking types
Figure S2: Encoder & decoder layer
Figure S1 illustrates five types of masking. In Sec. 5.3, we introduced two types of masking: anticipative masking and random masking. The anticipation mask M and the random mask M are defined for each masking, respectively. Additionally, we adopt the relation mask M and the boundary mask M introduced in DiffAct [43]. The relation mask M is proposed to help the model learn relational dependencies between actions. defined by M = (44 * 1), E (1, (1,2.... 2..,...)) and a E 2, ...,,,.... K). with 7' representing the number of frames and K denoting the number of action classes. Here, a specific action class is randomly selected for the mask. The boundary mask M hides visual features near the boundaries based on the soft ground truth B to manage the uncertainty associated with action transitions. Specifically, it is defined by M 1(B < 0.5), where E {1,2. T".
During training, masking type is randomly selected among five: mask M anticipative mask MA, random mask M relation mask M and boundary mask M for each iteration as detailed in Sec. 4.4. During inference, the mask M and the anticipative mask M utilized following the evaluation protocol [19, 65, 43, 20, 52, 25].
D Model architecture
We employ a modified version of ASFormer [65] utilized in DiffAct [43] as the baseline model. An encoder consists of E number of layers and decoder h consists of ND number of layers described in 4.c.4.2. Figure S2 illustrates detailed operations of the encoder and the decoder layers. An encoder layer consists of dilated convolution followed by dilated attention, instance normalization, and feed-forward network. Additionally, add relative position bias B attention scores to consider relative position relations among actions [48]. Similarly, decoder layer comprises dilated convolution followed by dilated attention, instance normalization, and feed-forward network. The output embedding E from the encoder is concatenated to the input of the decoder after dilated convolutions. Subsequently, they are embedded as queries and keys in the dilated attention operation. We refer the reader to [65, 43] for more details.
17

---

Table S1: Conditioning features
(a) Results on TAS
conditioning features
Fl @ [10, 25, 50)
Edit
Acc.
Avg.
Fr
85.6 / 83.6 74.8
79.8
82.8
81.3
E
91.6/ 90.7 84.8
86.0
89.3
88.5
A
90.5 / 89.4 83.2
84.6
87.6
87.1
(b) Results LTA
(a
0.2)
3 (a 0.3)
conditioning features
0.1
0.2
0.3
0.5
0.1
0.2
0.3
0.5
F
29.71
25.43
20.65
14.24
37.22
22.22
17.88
16.48
E
39.55
28.60
23.61
19.90
42.80
27.11
23.48
22.07
39.45
29.33
24.23
19.98
37,96
25.41
22.76
21.18
Table S2: Position embedding
(a) Results on TAS
position embedding
FI 10, 25, 50
Edit
Acc.
Avg.
none
90.0 89.7 84.3
85.3
88.5
Rel. bias
91.6 90.7 84.8
86.0
89.3
88.5
Rel. Emb.
90.5 89.6 83.9
84.7
88.3
87.4
Abs. Emb.
87.4/ 86.2 78.7
80.5
83.9
83.3
(b) Results on LTA
(a = 0.2)
3 (a 0.3)
position embedding
0.1
0.2
0.3
0.5
0.1
0.2
0.3
0.5
none
35.33
26.41
22.74
17.89
41.52
27.60
22.53
18.69
Rel. bias
39.55
28.60
23.61
19.90
42.80
27.11
23.48
22.07
Rel. Emb.
37.37
27.32
22.44
18.50
42.47
26.62
22.51
20.78
Abs. Emb.
35.91
26.84
22.48
17.70
30.02
20.87
18.24
17.71
Additional experiments
We provide additional experimental results, maintaining the same experimental settings as described in Sec.5.2 unless otherwise specified. The evaluation is conducted on the 50 Salads dataset.
Conditioning features. We investigate different types of conditioning features: masked features F'. output embeddings from the encoder, and the encoder prediction Á. Table S1 shows overall results, where using E is more effective than using other features. The effectiveness of utilizing the encoder our approach becomes apparent when comparing the first and second rows, where a significant performance drop is observed in the absence of the encoder. Since mask tokens replace visual tokens before going into the encoder, it is crucial for them to learn action relations through the encoder. Comparing the second and third rows, we also observe that utilizing features from intermediate layers yields slightly better results than using encoder predictions for both TAS and LTA.
Position embedding. In Table S2, we explore different types of position embeddings: relative position bias [48], relative position embedding [53], and absolute position embedding [15]. Comparing the first and second rows, we find that employing relative position bias enhances the overall performance TAS and LTA. From the second and third rows, we observe that relative position bias is also more effective than using learnable relative position embedding. Using absolute position embedding leads to decreased performance in both TAS and LTA. We find that learnable embeddings often cause overfitting problems during training. As result, adopt relative position bias in our model.
18

---

Table S3: Loss ablations
(a) Results on TAS
Lbd
Linoo
Lee
FI@ @ 10
F1@25
Fl @50
Edit
Acc.
Avg.
88.4
86.5
79.1
82.5
84.9
84.3
/
91.3
90.0
84.5
86.3
88.8
88.2
91.6
90.7
84.8
86.0
89.3
88.5
(b) Results on LTA
Lod
Lumo
Liee
3 (ar
0.2)
3 (ax 0.3)
0.1
0.2
0.3
0.5
0.1
0.2
0.3
35.62
27.04 04
20.17
15.93
34.38
22.33
19.96
16.94
39.19
28.99
23.13
19.45
39.53
25.19
22.67
V
39.55
28.60
23.61
19.90
42.80
27.11
23.48
22.07
Table S4: Effects of reconstruction loss Lrecon
(a) Results on TAS
Crewon
@ 25, 50)
Edit
Acc.
Avg.
85.6 / 83.6 74.8
79,8 86.0
82.8
81.3
91.6 / 90.7 84.8
89.3
88.5
(b) Results on LTA
Crecon
B (a = 0.2)
8 (a 0.3)
0.1
0.2
0.3
0.5
0.1
0.2
0.3
0.5
39.55
28.60
23.61
19.90
42.80
27.11
23.48
22.07
40.80
31.02
25.59
13.94
46.56
26.22
18.56
16.15
Loss ablations. We conduct ablation studies on the loss functions: boundary loss, smoothing loss, and cross-entropy loss. Table R4 presents results, demonstrating that the combination bounding loss and smoothing loss effective for both TAS and LTA. While the effectiveness of these losses in TAS is well-documented in previous rescarch [19, 43, 65], their impact on LTA has been less explored. Notably, the smoothing loss leads to significant performance gains in both tasks, indicating that smoothed predictions are beneficial.
Effects of reconstruction loss Crecon Masked outo-enccoing is technique used in training NLP models like BERT [15] and has recently been adapted to vision models [22, 28]. Inspired by this approach, we train our model to reconstruct input video features from the masked tokens as an auxiliary task. Specifically, we employ MLP layers on the encoder embeddings to reconstruct input video features and apply mean squared error (MSF) loss Lrezon during training.
Table S4 shows the overall results on both TAS and LIA tasks. In TAS, overall performance increases. We conjecture that reconstruction helps the model gain a deeper understanding of the underlying data structure and temporal dynamics by predicting the missing parts of the input. In LTA, we find that reconstruction is more effective relatively short-term anticipation. Since short-term predictions are often based on more immediate context, there is less uncertainty. As result, reconstructing masked features helps the model capture immediate patterns and transitions more accurately. However, for long-term predictions, as the model attempts to predict further into future, the uncertainty increases significantly. Long-term predictions involve more variables and potential changes, making them inherently less predictable. This increased uncertainty might cause performance degradation, making reconstruction less effective for action anticipation.
19

---

Table S5: Hyperparameters. We provide the hyperparameters used during training for each dataset.
hyperparameters
50 Salads
Breakfast
GTEA
of epochs batch size sample rate optimizer
5000
1000
10000
4
4
Adam
Adam
Adam
learning
0.000S
0.0001
0.0005
weight decay # encoder layers # of decoder layers dimension in encoder dimension in decoder
1e-06
10
12
10
8
8
64
256
64
24
128
24
conditioning features of E layers
7.97
7,,7.9)
{ 9
Ace Amo
0.5
0.5
0.5
0.025
0.1
Aga
0.0
0.0
0.0
Age
0.5
Adee
0.1
0.025
0.1
smo
Add
0.1
0.1
0.1
Datasets
The 50 Salads [58] dataset consists videos depicting 25 individuals preparing a salad. With over 4 hours of RGB-D video data, the annotations include 17 fine-grained action labels and 3 high-level activities. Notably, 50 Salads videos are longer than those the Breakfast dataset, averaging 20 actions per video. The dataset is partitioned into 5 splits for cross validations, and report the average performance across all splits. The Breakfast dataset is under the license of Creative Commons 4.0 International License.
The Breakfast [36] dataset consists of 1,712 videos featuring 52 individuals preparing breakfast in 18 different kitchens. Each video is assigned to one of the 10 activities associated with breakfast preparation, utilizing 48 fine- graiirr action labels to define these activities. The average video duration is approximately 2.3 minutes, encompassing around 6 actions per video. The dataset is divided into splits for cross-validations, and we report the average performance across splits. The Breakfast dataset is under the license of Creative Commons Attribution 4.0 International License.
The GTEA dataset [21] comprises 28 videos capturing 11 action classes related cooking activities. Each video includes 20 actions on average, and an average video duration is about minute. The dataset divided into 4 splits for cross-validations, and we report average performance across all splits. The GTEA dataset is under the license of Creative Commons Attribution 4.0 International License.
G Experimental details
Cross-task generalization. In Fig. 1-(),, we conduct experiments on cross-task evaluations with DiffAct [43], FUTR [25], and TempAgg [52]. To evaluate DiffAct on LTA, we limit the input video frames to FIOOT, with zero masks appended to future frame lengths concatenated to the encoder output embeddings, adhering to DiffAct's masking strategies. Subsequently, the decoder predicts future actions based on the observed video frames. To evaluate FUTR on TAS, utilize the encoder of FUTR as an action segmentation model. Note that we report the performance TempAgg on both TAS and LTA provided in the original paper [52].
Evaluation without using ground-truth prediction length on LTA. Baseline models of Cycle Cons. [20] and FUTR [25] anticipate future actions and their corresponding durations. The predicted durations are then directly multiplied by the ground-truth prediction length, 37, to generate the final predictions. In Table 6, we also experimented with using modified prediction length, ral", instead
20

---

of the ground -rrtt length. Please note that we use reproduced model for Cycle Cons. and the official model checkpoints for FUTR"
Implementation details. We provide additional implementation details to complement those de- scribed in Sec. 5.2. Table S5 presents the specific hyperparameters used in experiments for each dataset. All experiments are conducted on a single NVIDIA RTX-3080 GPU, We implement ActFusion using Pytorch [47] and some the official code repository of DiffAct [43] licensed under an MIT License.
H
Qualitative results
We provide additional qualitative results for both successful and failure cases in Fig. S3 and Fig. S4, respectively. Figure S3 demonstrates the promising results on both TAS and LTA, showing the efficacy joint learning these two tasks. the failure cases, figure S4a highlights the importance of accurate segmentation on anticipation, where inaccurate action segmentation of the observed frames leads to wrong future anticipation. In Fig. S4b, ActFusion fails to detect the 'take knife' action class in action segmentation. However, the model anticipates missing action classes that are not observed in the observed frames. This implies that the model can infer missing actions based on the action relations of the observed and unobserved actions.
I Discussion
In this paper, we have proposed a unified diffusion model for action segmentation and anticipation, dubbed ActFusion. We have demonstrated the effectiveness of the proposed method through compre- hensive experimental results, showing the bi-directional benefits of joint learning the two tasks. In this section, we will discuss the limitations, future work, and broader impact of the proposed method,
Limitations and future work. The proposed method shows the state-of-the-eeee the-art results on TAS across three benchmark datasets. However, the Trame-wise accuracy is slightly below compared to DiffAct [43]. We hypothesize that this discrepancy arises from the different masking strategies employed. In DiffAct, masking is applied the output embeddings of the encoder, ensuring that the encoder always fully observes the visual features from the entire video. However, in our approach, masking applied before the encoders, allowing for handling both visible and invisible tokens to unify the two tasks effectively. Consequently, the encoder may not always observe the entire visual features, potentially leading slightly lower accuracy. This issue could potentially be addressed by employing reconstruction methods for masked features, similar to masked autoencoders [28, 22]. By further training the model reconstruct the original features from the masked features, the encoder could be empowered to handle the masked features better. We leave this as our future work.
In future work, additional activity information could be integrated, particularly focusing on segment- wise action relations [8]. Moreover, exploring weakly-supervised learning approaches [39, 49, 37] could be beneficial for further enhancing the capabilities of method.
Broader impact. To the best of our knowledge, we are the first to integrate action segmentation and anticipation within unified model. We believe that our work lays the foundation for integrating these two tasks, offering substantial potential for real-world applications such as robots.
'Official model checkpoints of FUTR [25] are available at https://githutl httpp://ggttbbb..cmm//g///////1
'Officia code repository of DiffAct [43] is available at otttpp//ggthbb.co/////mmii/////
21

---

milk
Co
GT
TAS
LTA
anticipation siir milk
SIL
speces powder
pour milk
(a)
activity: cercals
Gl
TAS
LTA
observation
anticipation
S1L
take bowl
pour cereal
pour milk
activity: juice
GT
TAS
LTA
observation
anticipation
SII
QUE orange
squcoon orange
skko glass
pour jorrjsire
Figure S3: Qualitative results on successful cases
22

---

activity:
TAS
LTA
observation
anticipation
SIL
take cup
add teabag
pour
coffeee milk
(a)
activity: sandwich
GT
TAS
LTA
observation
anticipation
SIL
take knife
cut bun
smear butter
toppingentop
(b)
Figure $4: Qualitative results failure cases
23

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justifiicton right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]", ", it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS paper checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: Our main claims presented in the abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
24

---

Justification:
See Sec. I.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model vell-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification:
We provide theoretical explanation of diffusion models Sec. 3 and A.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We provide experimental details in Sec. 5 and Sec. G.
Guidelines:
The answer NA means that the paper does not include experiments.
25

---

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g.. with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibilityy In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We include the code and instructions for reproduction in the supplementary. The training and validation data is available online.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: /ttpsscccppblccc///ccpcpp gwwp://guuee.eeeeuu /CodeSubmisi asionPolicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
26

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification:
We specify all the training details in the main and supplementary.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification:
We do not report error bars due to the limited computational resources.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We include the amount and type of resources used for training in Sec. G.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
27

---

The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethies
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https:/ httppp:////uuips../.e cc cwtppcccppcceeeeeeeceecc ctwthi//eehhiisieeeleieleeeeee
Answer:
[Yes]
Justification:
We follow the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We provide the broader impact in Sec.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification:
We do not release data or models that can be highly misused.
Guidelines:
The answer NA means that the paper poses such risks.
28

---

Released models that have high risk for misuse or dual use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We cite all the papers that are relevant to the code, models, and datasets. We include the license of each asset in Sec. F and G.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URI
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Question:
Answer: [Yes]
Justification:
We include trained models in the supplementary for reproduction of our main
results.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Question:
Answer: [NA]

---

Justification:
We did not croudsource any information for this research.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
We did not croudsource any information for this research.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
30


Reviews:
[{"review_id": "PB1HGiamuU", "reviewer": "Reviewer_FW3D", "summary": "The author introduce a unified diffusion model for temporal action segmentation (TAS) and long-term action anticipation (LTA), dubbed ActFusion, where a single model is jointly trained to address these two problems effectively. A new anticipative masking is presented for the effective unification of two tasks, along with random masking to learn intra-action relations. ActFusion achieves the state-of-the-art performance on both TAS and LTA, demonstrating the effectiveness of joint learning of two tasks across standard benchmark datasets, 50 Salads and Breakfast, and GTEA.", "questions": "* The approach is currently limited to predicting action labels within a closed set. To extend this work to predict open-set action labels, what can be done?\n* While \"segmentation helps anticipation\" is evident from table 3, \"anticipation helps segmentation\" is considerably less significant in table 4. What's the reason behind this?\n* If further training the model to reconstruct the original features from the masked features, would it also improve the LTA task?\n* Paper writing\n    * What does circle and triangle means in figure 1(c)?\n    * In table2, why there are multiple underlined values (suppose to be second-highest value) in each column?", "limitations": "* Limitation\n    * The author notes the sub-optimal performance when using frame-wise accuracy to evaluate the TAS task and suggests a potential solution, though its effectiveness remains unproven. I am eager to see how these proposed adjustments might improve the model's performance.\n* Potential negative societal impact\n    * There is no potential negative societal impact of this work.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "* Originality\n    * The paper presents a approach to integrating two popular vision tasks, temporal action segmentation and long-term action anticipation, within a unified model. This is the first time these two problems have been investigated together in a single framework, highlighting the originality of the research. \n* Clarity\n    * The paper is clearly articulated and provides sufficient details. It offers an adequate background on diffusion model and how it is utilized. The inclusion of pseudo-code and actual code facilitates a better understanding for the reader. Additionally, the experimental settings are thoroughly described, which aids in the reproducibility of the results.", "weaknesses": "* The methodology lacks novelty. The model architecture, loss function (Cross-Entropy Loss, Temporal Smoothness Loss, Boundary Alignment Loss) and mask strategy (no mask, relation mask, boundary mask) closely resemble those used in the Diffusion Action Segmentation (https://arxiv.org/pdf/2303.17959v2). The primary distinction in this work is only its extension to include the long-term action anticipation task, and the introduction of anticipative masking.\n* The performance improvement relative to other state-of-the-art works is marginal. On TAS task, when compared to DiffAct, this model shows an approximate 1-point improvement across various metrics (F1, edit score, frame-wise accuracy) on different benchmarks (50 Salads, Breakfast, GTEA). \n* The model consistently achieves sub-optimal results when assessed using frame-wise accuracy as the metric on TAS task. As the authors point out, this could potentially be addressed by employing reconstruction methods for masked features. I am eager to see how these adjustments could enhance the model's performance.\n* More ablation study for LTA task could be included, e.g., How important is past context for the models (\\alpha) ? How far into the future can models predict (\\beta)? Table 2 reports only a limited range of settings. A more thorough analysis on these aspects would be highly valuable.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### **[Novelty]**\nPlease refer to the general response for our novelty.\n\n### **[Marginal performance]**\nWe would like to clarify that the performance improvements achieved by our model are significant. Figure R1 in the pdf file illustrates the performance of the Top 10 TAS models for each dataset listed in Table 1, based on their average performance across all metrics. The average performance gain between the adjacent models is 0.5 percentage points (pp), 0.7 pp, and 0.6 pp for the 50 Salads, Breakfast, and GTEA datasets, respectively.\nOur model, ActFusion, achieves performance gains of 0.7pp, 0.4pp, and 1.1pp compared to the second-best models for each dataset, and 1.1pp, 0.4pp, and 1.5pp compared to DiffAct. We believe the performance gains are meaningful, with notable improvements in two datasets.\n\n### **[Reconstruction of masked features]**\n\nMasked auto-encoding is a technique used in training NLP models like BERT [R1] and has recently been adapted to vision models [21, 27, 68]. Inspired by this approach, we train our model to reconstruct input video features from the masked tokens as an auxiliary task. Specifically, we employ MLP layers on the encoder embeddings to reconstruct the input video features and apply mean squared error (MSE) loss $L_{\\text{recon}}$ during training.\n\nTable R2 shows the overall results on both TAS and LTA tasks.  In TAS, overall performance increases. We conjecture that reconstruction helps the model gain a deeper understanding of the underlying data structure and temporal dynamics by predicting the missing parts of the input. In LTA, we find that reconstruction is more effective on relatively short-term anticipation. Since short-term predictions are often based on more immediate context, there is less uncertainty. As a result, reconstructing masked features helps the model capture immediate patterns and transitions more accurately. However, for long-term predictions, as the model attempts to predict further into the future, the uncertainty increases significantly. Long-term predictions involve more variables and potential changes, making them inherently less predictable.  This increased uncertainty might cause performance degradation, making reconstruction less effective for action anticipation.\n\n**[Table R2. Effects of reconstruction loss]**\n\n**(a) Results on TAS**\n|$L_{\\text{recon}}$ | F1@10 | F1@25 | F1@50 | Edit | Acc  | Avg. |\n|-|-|-|-|-|-|-|\n| -| 91.6|90.7|84.8|86.0|89.3|88.5|\n| ✓| 92.0|90.9|86.6|86.9|89.6|89.2|\n\n**(a) Results on LTA**\n|$L_{\\text{recon}}$|$\\alpha=0.2, \\beta=0.1$ | $\\alpha=0.2, \\beta=0.2$ | $\\alpha=0.2, \\beta=0.3$  | $\\alpha=0.2, \\beta=0.5$ | $\\alpha=0.3, \\beta=0.1$ | $\\alpha=0.3, \\beta=0.2$ | $\\alpha=0.3, \\beta=0.3$ | $\\alpha=0.3, \\beta=0.5$ |\n|-|-|-|-|-|-|-|-|-|\n|-| 39.55| 28.60| 23.61| 19.90| 42.80| 27.11| 23.48| 22.07|\n| ✓ | 40.80| 31.02| 25.59| 13.94| 46.56| 26.22 | 18.56| 16.15|\n\n[R1] J. Devlin et al.  BERT: Pre-training of deep bidirectional transformers for language understanding. In NAACL’19.\n\n### **[More analysis on LTA]**\nTable R3 shows the LTA performance across different observation ($\\alpha$) and prediction ($\\beta$) ratios. Average anticipation performance improves as the observation range increases. With more observations, uncertainty about future actions is relatively reduced, leading to more accurate predictions. Conversely, average anticipation performance decreases as the prediction range increases. Predicting further into the future presents more challenges due to increased uncertainty, as future actions become less predictable and more variable.\n\n**[Table R3. Analysis of observation and prediction ranges in LTA]**\n|| $\\alpha = 0.2$ | $\\alpha = 0.3$ | $\\alpha = 0.4$ | $\\alpha = 0.5$ | $\\alpha = 0.6$ | $\\alpha = 0.7$ | $\\alpha = 0.8$ | **Avg.** |\n|-|-|-|-|-|-|-|-|-|\n| $\\beta = 0.1$ | 39.56|42.81| 29.87| 37.07| 32.74| 27.53| 36.73| **35.2** |\n| $\\beta = 0.2$ | 28.6| 27.11| 25.41| 27.16| 27.68| 26.91| 42.28| **29.3** |\n| $\\beta = 0.3$ | 23.61| 23.48| 22.46| 24.50| 27.99| 29.94 | -| **25.3** |\n| $\\beta = 0.4$ | 22.71| 21.04| 22.44| 25.03| 31.08| -| -| **24.5** |\n| $\\beta = 0.5$ | 19.9| 22.07| 22.85| 28.08| -| -| -| **23.2** |\n| $\\beta = 0.6$ | 19.28| 22.78| 25.71|-|-| -| - | **22.6** |\n| $\\beta = 0.7$ | 19.93 | 25.62|-|-|-|-|-|**22.8**|\n| $\\beta = 0.8$ | 23.34|-|-|-|-|-|-| **23.3** |\n| **Avg.**| **24.6**| **26.4**| **24.8**| **28.4**| **29.9**| **28.1**| **39.5**||\n### **[Extension to the open-set action recognition]**\nWe appreciate the reviewer’s insight on extending our work towards open-set action recognition. To achieve this, we can use frozen image and text encoders from CLIP [R2] to obtain shared representations for actions and text embeddings. Similar to [R3], these embeddings can then be integrated into our model to enable open-set action recognition. We plan to explore this as a future direction for improving our approach.\n\n[R2] A. Radford et al. Learning transferable visual models from natural language supervision. In arXiv’21.  \n[R3] D. Chatterjee et al. Opening the vocabulary of egocentric videos. In Neurips’23. \n\n### **[Reasons: segmentation is more helpful on anticipation]**\nWe find that segmentation greatly enhances anticipation, while the effect of anticipation on segmentation is less significant (L271-272). Segmentation directly improves anticipation by providing accurate contextual cues and action boundaries of the observations, enabling the model to make more precise future anticipation. In contrast, anticipation helps segmentation more indirectly. Anticipation encourages the model to consider long temporal relations of actions within an activity, which may not result in immediate performance improvement in segmentation. \n\n### **[Explanation of Figure 1(c)]**\nPlease refer to the general response for a detailed explanation of Figure 1(c).\n\n### **[Multiple underlined values in Table 2]**\nThank you for pointing out. The underline of the performance of DiffAct on the 50 Salads dataset will be removed.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PbV9lJnx9F", "reviewer": "Reviewer_uA7n", "summary": "This paper proposes a new unified diffusion model called ActFusion, which solves the tasks of Time Action Segmentation (TAS) and Long Term Action Prediction (LTA) in a joint learning framework. To unify the two tasks, the model effectively handles the visible and invisible parts of the sequence during the training phase; The visible part is used for observed video frames, while the invisible part is used for future expectations. The experiment showed that the model achieved state-of-the-art performance in standard benchmark tests of 50 Salad, Breakfast, and GTEA.", "questions": "1. What is the difference between the Diffusion model used in the paper and the DiffAct model? They appear to be similar overall, despite using different Masks to unify the tasks of TAS and LTA.\n2. The paper employs so many loss functions for model training; the authors should analyze the impact of different losses rather than simply exploring the effects of the encoder and decoder.\n3. Since the denoising process of the diffusion model is a time-consuming task, I am interested in the computational efficiency of the model proposed in the paper. The authors are advised to provide information on the model's GFLOPs, parameter scale, and inference time.\n4. The authors need to further improve the interpretation of the figures, such as what the circles and triangles in Figure 1(c) represent.", "limitations": "In summary, the authors have creatively used different Masks to unify the tasks of TAS and LTA with a single model and achieved significant results. However, compared to existing models, there does not seem to be much modification, highlighting a disadvantage in innovation.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The design of the ActFusion model is novel, with its anticipative masking strategy and random masking method unifying the tasks of TAS and LTA.\n2. The model enhances its performance on both tasks through mutual promotion of TAS and LTA.\n3. The reported results in the paper surpass existing techniques on multiple evaluation metrics, demonstrating significant performance improvements.", "weaknesses": "1. What is the difference between the Diffusion model used in the paper and the DiffAct model? They appear to be similar overall, despite using different Masks to unify the tasks of TAS and LTA.\n2. The paper employs so many loss functions for model training; the authors should analyze the impact of different losses rather than simply exploring the effects of the encoder and decoder.\n3. Since the denoising process of the diffusion model is a time-consuming task, I am interested in the computational efficiency of the model proposed in the paper. The authors are advised to provide information on the model's GFLOPs, parameter scale, and inference time.\n4. The authors need to further improve the interpretation of the figures, such as what the circles and triangles in Figure 1(c) represent.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### **[Comparison with DiffAct]**\nPlease refer to the general response to the comparison with DiffAct.\n\n### **[Loss ablation studies]**\nWe conduct ablation studies on the loss functions: boundary loss, smoothing loss, and cross-entropy loss. Table R4 presents the results, demonstrating that the combination of bounding loss $L_{bd}$ and smoothing loss $L_{smo}$ is effective for both TAS and LTA. While the effectiveness of these losses in TAS is well-documented in previous research [18, 64, 42], their impact on LTA has been less explored. Notably, the smoothing loss leads to significant performance gains in both tasks, indicating that smoothed predictions are beneficial.\n\n**[Table R4. Loss ablation studies]**\n\n**(a) Results on TAS**\n| $L_{\\text{bd}}$ | $L_{\\text{smo}}$ | $L_{\\text{ce}}$ | F1@10 | F1@25 | F1@50 | Edit | Acc  | Avg. |\n|------------|------------|------------|-------|-------|-------|------|------|------|\n|            |            | &check; | 88.4  | 86.5  | 79.1  | 82.5 | 84.9 | 84.3 |\n|            | &check; | &check; | 91.3  | 90.0  | 84.5  | 86.3 | 88.8 | 88.2 |\n| &check; | &check; | &check; | 91.6  | 90.7  | 84.8  | 86.0 | 89.3 | 88.5 |\n\n**(a) Results on LTA**\n\n| $L_{\\text{bd}}$ | $L_{\\text{smo}}$ | $L_{\\text{ce}}$ | $\\alpha=0.2, \\beta=0.1$ | $\\alpha=0.2, \\beta=0.2$ | $\\alpha=0.2, \\beta=0.3$  | $\\alpha=0.2, \\beta=0.5$ | $\\alpha=0.3, \\beta=0.1$ | $\\alpha=0.3, \\beta=0.2$ | $\\alpha=0.3, \\beta=0.3$ | $\\alpha=0.3, \\beta=0.5$ |\n|------------|------------|------------|-------------|-------|-------|----------|-------|-------|--------|----------|\n|            |            | &check; | 35.62| 27.04 | 20.17 | 15.93| 34.38 | 22.33 | 19.96 | 16.94|\n|            | &check; | &check; | 39.19    | 28.99 | 23.13 | 19.45 | 39.53 | 25.19 | 22.67 | 19.88 |\n| &check; | &check; | &check; | 39.55 | 28.60 | 23.61 | 19.90    | 42.80 | 27.11 | 23.48 | 22.07 |\n\n### **[Computational efficiency]**\nTable R5 compares the computational cost of our model with ASFormer [64] for TAS, and FUTR [24] for LTA, in terms of the number of parameters, GPU memory usage during inference, and inference time. For TAS, as shown in Table R5 (a), although ASFormer has fewer parameters with lower inference time, it requires more GPU memory during inference and obtains lower performance. To improve computational efficiency, we reduce the DDIM inference steps to 10 and 1. This reduction decreases inference time while maintaining superior performance over ASFormer. \n\nFor LTA, as shown in Table R5 (b), our model is approximately eleven times smaller than FUTR, uses less GPU memory, but has a longer inference time. By reducing the DDIM inference steps to 1, our model achieves a similar inference time to FUTR.\nOverall, our model is practical and efficient since it can handle both TAS and LTA tasks with a unified model, eliminating the need for separate models and reducing GPU resource usage and the time required for separate training. Note that we use model checkpoints from the official GitHub repositories for all comparisons.\n\n**[Table R5. Computational efficiency]**\n\n**(a) Results on TAS**\n| method            | # inference steps | Avg. Performance | # parameters (M) | memory (GB) | inference time (s) |\n|-------------------|-------------------|------------------|------------------|-------------|--------------------|\n| AsFormer [64]         | 1                 | 81.9             | 1.134            | 0.272       | 1.66               |\n| ActFusion (ours)  | 1                 | 86.0             | 1.576            | 0.164       | 0.42               |\n| ActFusion (ours)  | 10                | 87.9             | 1.576            | 0.164       | 1.17               |\n| ActFusion (ours)  | 25                | 88.5             | 1.576            | 0.164       | 2.01               |\n\n**(a) Results on LTA**\n| method            | # inference steps | Avg. Performance | # parameters (M) | memory (GB) | inference time (s) |\n|-------------------|-------------------|------------------|------------------|-------------|--------------------|\n| FUTR [24]             | 1                 | 26.0             | 17.38            | 0.156       | 0.21               |\n| ActFusion (ours)  | 1                 | 26.2             | 1.576            | 0.151       | 0.26               |\n| ActFusion (ours)  | 10                | 27.8             | 1.576            | 0.151       | 1.04               |\n| ActFusion (ours)  | 25                | 28.4             | 1.576            | 0.151       | 2.14               |\n\n### **[Explanation of Figure 1(c)]**\nPlease refer to the general response for a detailed explanation of Figure 1(c).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AeRGvL4EqY", "reviewer": "Reviewer_zKDZ", "summary": "This paper extends DiffAct to perform both action segmentation and action anticipation. An anticipative masking with a learnable mask token is proposed. Experiments are conducted on the three common benchmark datasets.", "questions": "NA", "limitations": "The limitations were briefly discussed.", "rating": 6, "confidence": 1, "soundness": 1, "presentation": 3, "contribution": 3, "strengths": "1. The motivation for unifying action segmentation and action anticipation is reasonable, given the task similarity. It is also intuitive and reasonable to extend a generative framework from segmentation to anticipation, given the generative nature of anticipation.\n\n2. The learnable mask token is interesting.\n\n3. The ablation studies are relatively comprehensive.\n\n4. Codes are provided in the supplementary.", "weaknesses": "1. The major technical problem is that the proposed method assumes the ground truth video length is known for action anticipation at the test time. Without the ground truth video length T, the anticipative mask M^A can not be constructed during the inference time for anticipation. This is also shown in the code provided, where 'full_len' is input into the ddim inference function. This is problematic, conflicting with the goal of anticipation, and leading to incomparable experimental results.\n\n2. The technical novelty is limited. The main contribution is extending the DiffAct method with an anticipative mask, while other modules are from existing methods. But given the contribution of unifying the segmentation and anticipation tasks, this is not a deciding factor for me.\n\n3. It will be better to conduct experiments on Assembly101. Given the small data size and the saturated performance on GTEA, I would recommend a transition from GTEA to Assembly101 for this task.\n\nMinor:\n- In Figure 1c, what do the triangle and the circle mean?\n- In Table 1, some recent methods are missing, such as MVGA ICCV23 and RTK ICCV23.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### **[Inference setup in LTA]**\nWe would like to clarify that our model can predict future actions with an arbitrary length by adjusting the number of mask tokens needed for prediction; our model itself does not require a ground-truth length of anticipation, and the ground-truth length is used to \n generate the same length of anticipation for the convenience of evaluation in testing. Note that previous methods [19, 24, 43] also commonly used ground-truth video lengths during inference to generate the final predictions, which can be found in the released original codes. We obtain the codes for [19] from the authors.  We hope this clarifies the reviewer's concern. Otherwise, please let us know.\n\n\n### **[Novelty]**\nPlease refer to the general response for the novelty.\n\n\n### **[Experiments on Assembly101]**  \nDue to limited resources and time, we were unable to finish the full-scale experiment on time. In response to suggestions regarding Assembly 101,  we instead did a smaller scale experiment by randomly sampling 10% of the entire training set to train the models and evaluate the performance on the full validation set to ensure a fair comparison.\n\nTable R5 compares the performance of ActFusion and LTContext [5], the state-of-the-art TAS model on Assembly 101 from the experiment. We find that our model outperforms LTContext across all metrics, showing the potential advantages of the proposed method. We will provide a full-scale experiment in the final manuscript.\n\n**[Table R5. Experiments on Assembly101]**\n| method     | F1@10 | F1@25 | F1@50 | Edit | Acc  | Avg. |\n|------------|-------|-------|-------|------|------|------|\n| LTContext [5] | 18.7  | 15.9  | 11.2  | 17.6 | 27.3 | 18.2 |\n| ActFusion (ours) | 21.7  | 19.3  | 14.0  | 19.8 | 28.1 | 20.6 |\n\n### **[Explanation of Figure 1(c)]**\nPlease refer to the general response for a detailed explanation of Figure 1(c).\n\n### **[Missing references]**\nThank you for letting us know. We will include MVGA [R4] and RTK [R5] in our final manuscript.\n\n[R4] B. Jiang et al. RTK: Video action segmentation via contextually refined temporal keypoints. In ICCV’23.  \n[R5] N. Aziere and S. Todorovic. Markov game video augmentation for action segmentation. In ICCV’23.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all the reviewers for their insightful comments and suggestions. We are happy to see that the reviewers have given our work a positive evaluation, noting that “this is the first time these two problems have been investigated together in a single framework, highlighting the originality (FW3D)”, “the design of the AcFusion is novel (uA7n)”, “the model enhances its performance on both tasks through mutual promotion of TAS and LTA (uA7n)” and “it is also intuitive and reasonable to extend a generative framework from segmentation to anticipation, given the generative nature of anticipation (zKDz)”.

Nevertheless, the reviewers also point out important comments that:
1. the novelty of the proposed method should be explained,
2. revealing the effects of learning reconstruction of masked features for TAS is suggested, 
3. further analyses on LTA, loss functions, computational cost, and dataset are suggested. 

Through this rebuttal, we aim to clearly expose our novelty and provide further experimental results and analyses. We will revise the manuscript by incorporating the detailed comments from the reviewers.

In the general response, we address the questions posed by all reviewers regarding novelty and the explanation of Figure 1(c).

### **[Novelty]**
The primary novelty of the proposed method lies in unifying the two popular video tasks, temporal action segmentation (TAS) and long-term action anticipation (LTA). **The unification is not merely an extension but a novel framework that leverages the bi-directional benefits between TAS and LTA, maximizing synergies between these tasks.** None of the previous work [42, 24, 51] neither has introduced a single unified model to tackle the two tasks nor explored the bi-directional benefits. This integration is indeed crucial for practical applications, such as human-assistant robots, which need to recognize and anticipate future actions simultaneously.

**To achieve successful task integration, we introduce two types of masking strategies: anticipative masking ($M^\texttt{A}$) and random masking ($M^\texttt{R}$).** Anticipative masking plays a crucial role in effective task integration and random masking leverages to learn intra-action relations from a video. **However, simply incorporating these masking strategies does not necessarily guarantee the optimal performance for both TAS and LTA.** In Table R1, we applied anticipative and random masking to DiffAct by replacing visual embeddings with zero vectors after encoder processing and using them as conditions for the diffusion process in the decoder.
The results in Table R1 show that these masking strategies improve TAS performance but remain below existing state-of-the-art models for LTA [51, 19, 25, 24, 65], 

We hypothesize that this is likely due to the limited information from the zero vectors used in future anticipation, which does not fully leverage the information from the visible tokens. To address this, **we propose a learnable masking strategy, where input visual features are replaced with learnable mask tokens provided to the encoder.** These tokens are trained to learn temporal relations between visible and invisible parts through attention mechanisms. **In our model, both the encoder and decoder are trained to handle visible and invisible parts for effective task unification.** The introduced masking strategy maximizes synergies between the two tasks, leading to achieving state-of-the-art performance in both TAS and LTA in Tables 1 and 2.  We believe our approach presents a novel integrative framework for unifying the two tasks by introducing an effective learnable masking strategy with two types of masking. 

**[Table R1. Effects of a learnable masking strategy]**

(a) **Results on TAS**

| method   | $M^\texttt{A}$ | $M^\texttt{R}$ | F1@10 | F1@25 | F1@50 | Edit | Acc  | Avg. |
|---------|-------|-------|-------|-------|-------|------|------|------|
| DiffACT | -     | -     | 90.1  | 89.2  | 83.7  | 85.0 | 88.9 | 87.6 |
| DiffACT | ✓     | ✓     | 91.1  | 89.8  | 84.1  | 85.9 | 88.9 | 87.9 |
| ActFusion (ours)    | ✓     | ✓     | 91.6  | 90.7  | 84.8  | 86.0 | 89.3 | 88.5 |

(a) **Results on LTA**

| method   | $M^\texttt{A}$ | $M^\texttt{R}$ | $\alpha = 0.2$ $\beta = 0.1$ | $\alpha = 0.2$ $\beta = 0.2$ | $\alpha = 0.2$ $\beta = 0.3$ | $\alpha = 0.2$ $\beta = 0.5$ | $\alpha = 0.3$ $\beta = 0.1$ | $\alpha = 0.3$ $\beta = 0.2$ | $\alpha = 0.3$ $\beta = 0.3$ | $\alpha = 0.3$ $\beta = 0.5$ |
|---------|-------|-------|-----------------------------|-----------------------------|-----------------------------|-----------------------------|-----------------------------|-----------------------------|-----------------------------|-----------------------------|
| DiffACT | -     | -     | 11.8                        | 11.3                        | 11.3                        | 10.7                        | 20                          | 17.2                        | 16.5                        | 16.6                        |
| DiffACT | ✓     | ✓     | 30.3                        | 27.0                        | 19.1                        | 11.3                        | 37.4                        | 22.1                        | 15.6                        | 13.0                        |
| ActFusion (ours)   | ✓     | ✓     | 42.8                        | 33.9                        | 26.0                        | 20.7                        | 43.1                        | 25.8                        | 21.3                        | 20.7                        |

### **[Explanation of Figure1(c)]**

We apologize for not providing detailed explanations for Figure 1(c). In this figure, the circles represent the main tasks the models are proposed to address, while the triangles indicate auxiliary tasks used during training but not evaluated. We will include the descriptions in the final manuscript.

Meta-review: All 3 reviewers were inclined to accept the paper. All reviewers agree the paper tackles an important problem in an interesting way and the results show that the method improves over previous SOTA significantly. The AC has read all reviews and, aligned with the reviewer recommendation, the AC is recommending the paper be accepted to NeurIPs.
Decision: Accept (poster)