Title: ActFusion: a Unified Diffusion Model for Action Segmentation and Anticipation
Venue: NeurIPS 2024
Accepted: yes
Keywords: temporal action segmentation

long-term action anticipation
Abstract: Temporal action segmentation and long-term action anticipation are two popular vision tasks for the temporal analysis of actions in videos. 
Despite apparent relevance and potential complementarity, these two problems have been investigated as separate and distinct tasks. In this work, we tackle these two problems, action segmentation, and action anticipation, jointly using a unified diffusion model dubbed ActFusion. 
The key idea to unification is to train the model to effectively handle both visible and invisible parts of the sequence in an integrated manner;
the visible part is for temporal segmentation, and the invisible part is for future anticipation. 
To this end, we introduce a new anticipative masking strategy during training in which a late part of the video frames is masked as invisible, and learnable tokens replace these frames to learn to predict the invisible future.
Experimental results demonstrate the bi-directional benefits between action segmentation and anticipation.
ActFusion achieves the state-of-the-art performance across the standard benchmarks of 50 Salads, Breakfast, and GTEA, outperforming task-specific models in both of the two tasks with a single unified model through joint learning.

ActFusion: a Unified Diffusion Model for Action Segmentation and Anticipation

Abstract

Temporal action segmentation and long-term action anticipation are two popular vision tasks for the temporal analysis of actions in videos. Despite apparent relevance and potential complementarity, these two problems have been investigated as separate and distinct tasks. In this work, we tackle these two problems, action segmentation and action anticipation, jointly using a unified diffusion model dubbed ActFusion. The key idea to unification is to train the model to effectively handle both visible and invisible parts of the sequence in an integrated manner; the visible part is for temporal segmentation, and the invisible part is for future anticipation. To this end, we introduce a new anticipative masking strategy during training in which a late part of the video frames is masked as invisible, and learnable tokens replace these frames to learn to predict the invisible future. Experimental results demonstrate the bi-directional benefits between action segmentation and anticipation. ActFusion achieves the state-of-the-art performance across the standard benchmarks of 50 Salads, Breakfast, and GTEA, outperforming task-specific models in both of the two tasks with a single unified model through joint learning.

1 Introduction

In everyday life, when interacting with people, we anticipate their future actions while recognizing their actions observed in the present and the past. Similarly, for effective human-robot interaction, robotic agents must recognize ongoing actions while anticipating future behaviors. Two essential tasks in computer vision for such a temporal understanding of human actions are temporal action segmentation (TAS) [38, 19, 32, 61, 12, 65, 3, 43, 63, 8] and long-term action anticipation (LTA) [2, 34, 25, 45]. The task of TAS aims at translating observed video frames into a sequence of action segments, while the goal of LTA is to predict a plausible sequence of actions in the future based on the observed video frames. These tasks are closely related in terms of understanding the relations between actions; recognizing actions in the present and the past may improve anticipating action in the future, and the ability to anticipate the future may also enhance recognizing observable actions when facing visual ambiguities.

Despite the apparent relevance and potential complementarity, these two problems have been investigated as separate and distinct tasks. While a growing body of work has shown remarkable progress on both TAS and LTA, these methods are primarily designed for one task (see Fig. 1a) and show poor generalization when applied to the other (e.g., see FUTR [25] and DiffAct [43] in Fig. 1c). Some prior methods tackle both tasks simultaneously, but they rely on task-specific architectures and require separate training for each individual task (e.g., see TempAgg [52] in Fig. 1c).

In this paper, we introduce ActFusion, a unified diffusion model that addresses TAS and LTA through a single architecture and training process, as illustrated in Fig. 1b. The core idea of our unification lies in training the model to effectively handle two different parts of the sequence: visible parts for action segmentation and invisible parts for action anticipation. Accordingly, we propose a

(a) task-specific models

TAS model

?

LTA model

Seg.

observation

observation

anticipation

Ant.

ActFusion

?

Ant.

Seg.

Seg.

observation

observation

anticipation

(b) ActFusion (ours)

(c) evaluation on TAS & LTA

DiffAct

FUTR

TempAgg

Ours

TAS

LTA

75.5

75.1

54.3

12.6

14.1

26.6

23.1

28.5

new anticipative masking strategy in which a late part of the video frames is masked as invisible, and learnable tokens replace it. We also introduce a random masking strategy to help the model accurately identify actions when some parts of the video are ambiguous [59], drawing inspiration from previous methods [15, 6, 28, 59]. ActFusion generates action sequences from Gaussian noise through iterative denoising, conditioned on both visual features and mask tokens. Through this process, the model jointly learns to classify action labels of visual features and predict future actions of mask tokens. As shown in Fig. 1c, ActFusion achieves superior performance in both action segmentation and anticipation, demonstrating stronger cross-task generalization compared to previous methods [43, 25, 52].

Furthermore, we find that the benchmark evaluation of some prior LTA methods [2, 20, 25] has exploited the ground-truth length of input videos in testing time, which makes the evaluation unrealistic and problematic since the duration of future actions is supposed to be unknown in advance. To address this issue, we conduct comprehensive experiments both with and without ground-truth length information, providing insights into more realistic deployment scenarios.

Our contribution can be summarized as follows: 1) We introduce ActFusion, a unified diffusion model that jointly addresses TAS and LTA through a single training process. 2) We present anticipative masking for effective task unification, along with random masking to enhance robustness against visual ambiguities. 3) Comprehensive experimental results show that jointly learning both tasks provides bi-directional benefits. ActFusion achieves state-of-the-art performance on TAS and LTA across benchmark datasets - 50 Salads and Breakfast, and GTEA - demonstrating the effectiveness of the proposed method.

2 Related work

Temporal action segmentation (TAS). The goal of TAS [19, 65, 43, 63, 12, 23, 5, 8, 32, 3, 16] is to classify frame-wise action labels in a video. Earlier approaches employ temporal sliding windows [50, 33] for action segment detection, grammar-based methods [37, 36] have been introduced to incorporate a temporal hierarchy of actions during segmentation. Temporal convolution networks [38, 19] and Transformer-based models [65, 8] are introduced based on deep learning methods. Since learning long-term relations of actions from activity videos is challenging, a series of work has been proposed to develop refinement strategies [30, 32, 23, 12, 57, 63, 3, 24] that can be applied to the TAS models [19, 65]. Recently, DiffAct [43] is proposed to iteratively denoising action predictions conditioned on the input video features adopting the diffusion process. In a similar spirit, the proposed ActFusion is based on the diffusion process, focusing on unifying TAS and LTA through anticipative masking. We demonstrate the bidirectional benefits between the two tasks, showing that learning segmentation along with anticipation is effective.

Long-term action anticipation (LTA). LTA [2, 1, 34, 20, 52, 25, 44, 26] has recently emerged as a crucial task for predicting a sequence of future actions in long-term videos. Initial models use RNNs

and CNNs [2], while time-conditioned anticipation [34] introduces one-shot anticipation of specific future timestamps. A GRU-based model [20] used cycle-consistent relations of past and future actions. Sener et al. [52] proposed TempAgg, a multi-granular temporal aggregation method for action anticipation and recognition, utilizing different model architectures and task-specific losses for the two tasks. Gong et al. [25] propose a transformer model for parallel anticipation, dubbed FUTR, empirically showing that learning action segmentation as an auxiliary task is helpful for anticipation. Nawhal et al. [44] propose a two-stage learning approach for LTA and Zhang et al. [66] present object-centric representations using visual-language pre-trained models for LTA. While previous work [52, 20, 25, 44] adopts TAS as an auxiliary task to help learn LTA, a unified model evaluated on both tasks is rarely explored, often showing poor cross-task generalization performance (Fig. 1c). The exception is TempAgg, which requires separate training and model architecture for the two tasks. In this work, we present a unified model that tackles both TAS and LTA in a single training process.

Diffusion models. Recent success in denoising diffusion models [55, 29, 56] opens a new era of computer vision research. The diffusion models learn the original data distribution through the iterative denoising process. Diffusion models have recently shown notable success in various domains, such as image generation [51, 14], video generation [42, 27, 64], object detection [13], semantic segmentation [7], temporal action segmentation [43], self-supervised learning [62], and etc [18]. Recently, DiffMAE [62] integrates masked autoencoders with the diffusion models, where the model learns to denoise masked input while learning data distributions through generative pre-training of visual representations. In this work, we present a unified diffusion model that effectively integrates TAS and LTA through masking, where the model learns to denoise action labels conditioned on both visual and mask tokens. In this way, the model can effectively learn temporal relations between actions by classifying visual tokens and inferring missing actions of the mask tokens.

3 Preliminary

In this section, we give a brief overview of diffusion models [29, 54]. The diffusion models learn a data distribution by mapping and denoising noises from the original data distribution. The training process of diffusion models involves forward and reverse processes from random noise. During the forward process, Gaussian noise is added to the original data, while in the reverse process, a neural network learns to reconstruct the original data by iteratively removing noise.

The forward process, or diffusion process, transforms the original data x 0 into noisy data x s :

Here, a noise ϵ ∼ N (0 , I ) is added to the original data distribution x 0 following the decreasing function γ ( s ) of time step s ∈ { 1 , 2 , ..., S } , where S represents the entire forward time-step. Note that the function γ ( s ) determines the intensity of the noise added to the original data following the pre-defined variance schedule [29].

In the reverse process, a neural network f ( x s , s ) is trained to recover the original data x 0 from noisy data x s using l 2 regression loss:

During inference, the model f iteratively denoises pure noise x S to reconstruct the original data x 0 , i.e. , x S → x S -∆ → ... → x 0 , following the updating rule [29, 54]. We refer the reader to [29, 54] and Sec. A for more details.

In our context, the neural network learns to generate action sequences from Gaussian noise, conditioned on visible features for action segmentation and mask tokens for action anticipation.

4 Proposed approach

We present ActFusion, a unified diffusion model for action segmentation and anticipation. This section describes the problem setup in Sec. 4.1, the model architecture in Sec. 4.2, and the proposed masking strategies and training objectives in Sec. 4.3 and Sec. 4.4, respectively.

Inference

Training

0

𝑇

enc

𝑔

Conditioning

dec

ℎ

# 𝑨

!

𝑨

"

0

𝑇

Masking

ant

random

none

boundary

rel

[M]

[M]

[M]

: mask tokens

[M]

: visual tokens

𝑭

𝑬

𝑭′

•

Training:

all masks

•

Inference

:

none

(

TAS

),

ant

(

LTA

)

dec

ℎ

# 𝑨

"#∆

# 𝑨

"

0

𝑇

iterative denoising

[M]

[M]

[M]

[M]

[M]

[M]

[M]

[M]

[M]

[M]

[M]

[M]

4.1 Problem setup

Temporal action segmentation (TAS) aims to classify input video frames into a sequence of predefined action classes, while long-term action anticipation (LTA) predicts future actions based on partially observed video sequences. Formally, given a video sequence F = [ F 1 , F 2 , · · · , F T ] of length T , TAS predicts frame-wise action labels A = [ A 1 , A 2 , · · · , A T ] , where each A i is a one-hot vector representing the action class. In LTA, given the first N O = ⌈ αT ⌉ observed frames, the goal is to anticipate action labels for the subsequent N A = ⌈ βT ⌉ frames, where α ∈ [0 , 1] and β ∈ [0 , 1 -α ] represent the observation and anticipation ratios, respectively. Here, ⌈·⌉ denotes the ceiling function.

4.2 ActFusion

ActFusion aims to unify TAS and LTA by leveraging an encoder-decoder architecture with adaptive masking strategies. Figure 2 illustrates the overall pipeline, where ActFusion consists of a masked encoder g and a denoising decoder h . The encoder obtains visual features and mask tokens as input and generates embedded tokens as output. The decoder then progressively reduces noises through an iterative denoising process conditioned on the embedded tokens.

During training, we randomly sample a time step s ∈ { 1 , 2 , ..., S } for each iteration and add noise ϵ ∼ N (0 , I ) to the ground-truth action labels A 0 following Eq. 1, resulting in noisy action labels A s . The decoder then aims to denoise A s to reconstruct the original action labels ˆ A 0 . During inference, the decoder starts with Gaussian noise ˆ A S and progressively denoises it following the DDIM [54] update rule to generate final predictions ˆ A 0 , i.e. , ˆ A S → ˆ A S -∆ → ... → ˆ A 0 .

The key to our unified approach lies in training the model to effectively process both visible and invisible parts of the sequence, where the visible part corresponds to observed video frames and the invisible part represents future frames to be anticipated. To this end, we introduce anticipative masking that replaces unobserved frames with learnable mask tokens, enabling the model to learn future predictions. We apply this masking strategy consistently during both training and inference to achieve joint learning of action segmentation and anticipation. We further incorporate random masking, where video frames are randomly masked to enhance robustness against visual ambiguities. Both masking strategies are detailed in Sec. 4.3.

Input structuring. Given input video features F ∈ R T × C with T frames of C feature dimensions, we start by defining a binary mask matrix M ∈ { 0 , 1 } T × 1 . This mask serves as a frame selector, where a value of 0 indicates a frame to be masked and replaced by mask tokens, while a value of 1 indicates an unmasked frame. The learnable mask token, denoted as m ∈ R 1 × C , replaces the visual

features in frames selected for masking, producing the input features F ′ for the encoder:

where ⊙ denotes element-wise multiplication and 1 i × j represents a matrix of ones with dimensions i × j . Here, M and m are broadcasted along the channel and temporal dimensions, respectively.

For our model architecture, we use a modified version of ASFormer [65] used in DiffAct [43], which employs dilated attention to capture both local and global relations in the input sequence (See Fig. S2 for the detailed model architecture).

Encoder. The encoder consists of the N E layers, each consisting of a dilated 1-d convolution followed by instance normalization, dilated attention, and a feed-forward network [65]. In the dilated attention mechanism, the receptive field is limited to a local window size w = 2 i for the i -th layer, where the increasing dilation captures progressively broader temporal relations. The output of each layer is combined with its input via a residual connection before proceeding to the subsequent layer. Given the input features F ′ , the encoder g produces embedded tokens E ∈ R T × D as:

where D represents the dimensions of embedded tokens. A fully-connected (FC) layer W enc ∈ R D × K is applied to E to obtain frame-wise classification logits from the encoder followed by a softmax:

where K is the number of action classes and σ is the softmax.

Decoder. The decoder is composed of N D sequential layers. Each layer consists of dilated 1-d convolution, dilated attention, instance normalization, and feed-forward networks. As in the encoder, the dilation ratio for the i -th layer is set to w = 2 i . The output of each layer is combined with its input through a residual connection before being passed into the subsequent layer.

Given a time step s , noisy action labels A s at time step s , and encoder embeddings E , the decoder h produces output embeddings Y ∈ R T × D according to:

The final action label predictions ˆ A 0 ∈ R T × K are obtained by projecting Y through a fully-connected layer W dec ∈ R D × K followed by softmax:

4.3 Masking strategy

We introduce two distinct masking strategies: anticipative masking and random masking.

Anticipative masking. Anticipative masking enables joint learning of TAS and LTA by separating visible and invisible parts of the input sequence. This strategy masks the latter portion of video frames, requiring the model to predict future actions based on observed frames. Given a video length T , we define a binary anticipation mask M A ∈ { 0 , 1 } T that sets visible frames to 1 and invisible frames to 0: M A i = ✶ ( i ≤ N O ) , where ✶ is an indicator function and N o represents the number of observed frames. Unlike causal masking [60] that prevents attending to future tokens, our anticipative masking allows interactions among visible tokens while maintaining a clear boundary for anticipation.

Random masking. Random masking aims to provide robustness in prediction when parts of video frames are missing or ambiguous [59]. A video is first divided into pre-defined clips of size Q , resulting in N P = ⌈ T Q ⌉ total clips. Then, N R clips are randomly selected to be masked. A binary random mask is defined by M R i = ✶ ( ∃ j ∈ P , ( j -1) Q < i ≤ jQ ) , where P is a randomly selected subset of { 1 , · · · , N P } with | P | = N R .

For fully observable scenarios, we utilize a no mask strategy M N where all frames remain visible. Additionally, we adopt two masking strategies [43], the relation mask M S and the boundary mask M B , as illustrated in Fig. 2. The relation mask M S randomly masks segments associated with an action class to learn inter-action dependencies. The boundary mask M B masks frames at action transitions to enhance boundary detection. See Sec. C for the detailed formulations of the masks.

During training, one of five masking strategies is randomly selected: no mask M N , anticipative mask M A , random mask M R , relation mask M S , and boundary mask M B . For inference, no mask M N is used for TAS and anticipative mask M A for LTA.

4.4 Training objective

The model is trained with three types of losses: cross-entropy loss L ce for frame-wise classification, boundary smoothing loss L smo [19], and boundary alignment loss L bd [43]. These losses are applied to both encoder and decoder, where the encoder serves as an auxiliary task to enhance discrimination ability. Given the ground-truth action label A 0 ∈ R T × K and the predictions ˆ A ∈ R T × K , the cross entropy loss is defined by:

To prevent over-segmentation errors, a temporal smooth loss [19] between adjacent frames based on a truncated mean squared error over the frame-wise log probabilities are defined by:

where the difference of the log probabilities of two adjacent frames is truncated with a threshold value. For precise boundary detection, the boundary alignment loss [43] is employed based on the binary cross-entropy loss:

̸

where ¯ B = κ ( B ) represents a softened version of the ground-truth action boundary B i = ✶ ( A 0 i = A 0 i +1 ) , achieved through a Gaussian kernel κ . The encoder and decoder losses are defined by

where λ denotes a scaling factor. The total loss is defined as L total = L enc + L dec .

5 Experiments

In this section, we conduct experiments to demonstrate the effectiveness of our model. All reported experimental results are obtained from inference using a single unified model. We evaluate our

method on three widely-used benchmark datasets: 50 Salads [58], Breakfast [36], and GTEA [21] (see Sec. F for details). All three datasets are used to evaluate on TAS, while 50 Salads and Breakfast are used for evaluating LTA, following the protocols of the previous work [19, 65, 43, 20, 52, 25].

Evaluation metrics. For evaluation metrics for TAS, we report F1@ { 10 , 25 , 50 } scores, the edit score, and frame-wise accuracy [19, 65, 43]. The F1 scores and the edit score are segment-wise metrics, and accuracy is a frame-wise metric. Mean over classes accuracy (MoC) is adopted as an evaluation metric for LTA [2, 34, 52, 25].

Implementation details We utilize the pre-trained I3D features [9] as input video features for all datasets provided by [19]. For the diffusion process, we set the entire time step S as 1000 [29, 54], with a skipped time step for inference set to 25 [54]. For the anticipation mask M A , we set the observation ratio α ∈ { 0 . 2 , 0 . 3 , 0 . 4 , 0 . 5 , 0 . 6 , 0 . 7 , 0 . 8 } . For the random mask M R , we fix the size of patch w to 10 for both tasks and randomly select the number of masked patches N R to 25, 10, and 20 for 50 Salads, Breakfast, and GTEA, respectively. For the encoder and decoder, we adopt a modified version of ASFormer [65] using relative positional bias where the maximum number of neighbors is set to 100. During inference, we set α to 1 and β to 0 for TAS [65, 19, 43]. For LTA, we set α ∈ { 0 . 2 , 0 . 3 } and β ∈ { 0 . 1 , 0 . 2 , 0 . 3 , 0 . 5 } , following the evaluation protocols [2, 20, 52, 25]. See Sec. G for more details.

5.1 Comparison with the state of the art on TAS and LTA

Tables 1 and 2 present performance comparisons across benchmark datasets, where our single unified model achieves superior results on both tasks. Table 1 shows the results on TAS, where ActFusion outperforms all task-specific models demonstrating the benefits of joint learning. Table 2 compares LTA performance across different datasets and input types: predicted action labels of the visual features [49] and pre-trained I3D features [9]. Overall, ActFusion achieves state-of-the-art performance on two benchmark datasets, demonstrating the efficacy of joint learning for TAS and LTA based on the diffusion process. Note that we do not include the performance of ANTICIPATR [44] due to differences in evaluation setup, as reported in [67].

5.2 Analysis

We conduct comprehensive analyses to validate the effectiveness of the proposed method. In the following experiments, we evaluate our approach on the 50 Salads dataset. All experimental settings are the same as explained in Sec. 5 unless otherwise specified.

(a) Results on TAS

(b) Results on LTA

Segmentation helps anticipation. To validate the impact of learning TAS on LTA, we conduct loss ablation experiments by removing the action segmentation loss on the observed N O frames. Table 3 shows the results. Specifically, we exclude the encoder loss L enc in Eq. 11, while omitting the decoder loss L dec in Eq. 12 on the observed frames. For simplicity, we denote the decoder loss for the observed frames and unobserved frames to be anticipated as L dec O and L dec A , respectively. Note that L dec O are utilized for effective anticipation learning. By comparing (1) and (3) in Table 3, we observe a significant performance drop when removing the loss on the observed frames. We observe that applying segmentation loss on the observed frames in the decoder improves performance when comparing (1) and (3). The overall results show that learning action segmentation plays a crucial role in improving action anticipation.

Anticipation helps segmentation. To validate the effect of jointly learning anticipation along with segmentation, we conduct ablation studies on anticipative masking M A in Table 4. Comparing (1) and (2) in Table 4a, we find that using anticipation masking helps improve the performance of TAS. In comparison to the results in Table 4, where segmentation greatly enhances anticipation, the performance improvement without using anticipation masking is slightly lower. Nevertheless, we find that anticipation does contribute positively to the overall metrics of segmentation.

Ablation studies on masking types . To evaluate the effects of different types of masking on TAS and LTA, we conduct ablation studies in Table 4. The performance on TAS and LTA are presented in Table 4a and 4b, respectively. We observe that anticipative masking M A plays a significant role in the joint learning of TAS and LTA, as evidenced by the substantial performance drop in LTA when comparing row (2) with the other rows in Table 4b. Random masking M R significantly reduces performance in both TAS and LTA, as shown in row (3) of Table 4. Furthermore, the use of boundary masking M B and relation masking M S is also essential for both tasks. Due to the limited space, we only report the performance on LTA when the observation ratio α is set to 0.3.

Effects of learnable mask tokens. To verify the effects of learnable mask tokens m , we replace m with zero vectors. Table 5 shows the overall results. Comparing rows (1) and (2) in Table 5, we find that using learnable mask tokens is more effective for both TAS and LTA. We conjecture that this is due to the mask tokens being embedded with the visual tokens within the encoder, which aids in effectively anticipating the invisible parts.

Position of mask tokens. Instead of providing mask tokens as input to the encoder, we conduct experiments on providing mask tokens as input to the decoder, as shown in row (3) in Table 5. In this experiment, only the visible frames are given as input to the encoder, while mask tokens m ′ ∈ R 1 × D are applied in the decoder to fill the original masked positions, similar to the approach used in masked auto-encoder [28]. By comparing rows (2) and (3) in Table 5, we observe that using mask tokens in the encoder is more effective than using them in the decoder. We hypothesize that embedding mask tokens alongside visual tokens within the encoder benefits joint learning of TAS and LTA. Furthermore, when mask tokens are provided only to the decoder, they do not receive the encoder loss L enc , which may negatively impact performance.

(b) Results on LTA

87.8

88.5

86.9

86.6

10

25

50

100

Avg. performance

The number of masked clips

TAS

27.2

28.4

27.6

26.9

10

25

50

100

Avg. performance

The number of masked clips

LTA

𝑁 R

𝑁 R

86.0

87.9

88.5

88.3

1

10

25

50

Avg. performance

The number of inference steps

TAS

26.2

27.8

28.4

28.5

1

10

25

50

Avg. performance

The number of inference steps

LTA

The number of masked clips N R in random masking. To determine the number of masked clips N R in random masking, we conduct experiments by adjusting N R from 10 to 100 while fixing the window size Q of each masked clip N P to 10. Figure 3 illustrates the result, suggesting that employing an appropriate amount of masking is crucial.

The number of inference steps in the diffusion process. We compare performance according to the number of inference steps of the diffusion process in Fig. 4. We observe consistent performance increases as the number of inference steps increases. As the increasing step requires more computation, we choose to use 25 steps for inference in all of our experiments.

5.3 Qualitative results

Figure 5 presents qualitative results evaluated on both TAS and LTA using a single model. Each figure includes video frames, ground-truth action sequences, and predicted results for TAS and LTA. Only the visible parts (observed frames) are used as input during inference on LTA. Overall results show that ActFusion effectively handles both visible and future segments, accurately classifying current actions and anticipating future ones. Additional results are provided in Fig. S3 and S4.

5.4 Evaluation without using ground truth prediction length on LTA

Benchmark evaluations in previous work, including those of [20, 25] and our results presented above, have been typically conducted following the evaluation protocol of [2] where prediction length N A is set to βT in testing; β and T are the prediction ratio and the ground-truth video length, respectively. This can be seen as a leakage of ground-truth information in testing because the exact length of future actions is supposed to be inherently unknown during inference in real-world scenarios. In this subsection, we thus rectify the evaluation setting by determining the prediction length as rN O based solely on the number of observed frames N O , where r is a hyperparameter that adjusts the relative length of future predictions. We then train our method with this modified anticipation masking strategy, denoted as ActFusion † . In this experiment, r is set to 4.

Using the rectified evaluation protocol, we compare ours with Cycle Cons. [20] and FUTR [25], whose codes are available 1 . For a fair comparison, we modify both models to use the same prediction length rN O , denoted as Cycle Cons. † and FUTR † . The results are summarized in Table 6 where the numbers in parentheses indicate relative performance changes compared to those in Table 2. Table 6 shows that all the methods exhibit overall performance degradation when ground-truth length information is not used. These observations reveal that previous results have benefited from the use of ground-truth length T , leading to information leakage in testing. Therefore, we suggest that future research avoid relying on this information for a more realistic and fair comparison. On the other hand, the results also show that our approach consistently outperforms the methods across benchmark datasets, demonstrating its effectiveness even without ground-truth prediction length information. Please refer to Sec. G for further details of the experimental setup.

SUP

BF, Split1, P07_stereo01_P07_friedegg activity: fried egg

GT

SIL

butter pan

add saltnpepper

fry egg

put egg2plate

crack egg

TAS

LTA

observation

(a) Breakfast

GT

TAS

LTA

action start

cut tomato

cut cheese

place cheese into bowl

cut lettuce

peel cucumber

place lettuce into bowl

add vinegar

place cucumber into bowl

mix ingredients

add oil

activity: making a salad

cut cucumber

add salt

add pepper

serve salad onto plate

add dressing

action end

mix dressing

observation

rgb-25-2

anticipation

(b) 50 Salads

dataset

methods

β

(

α

= 0

.

2)

β

(

α

= 0

.

3)

0.1

0.2

0.3

0.5

0.1

0.2

0.3

0.5

50 Salads

†

[20]

31.85 (2.91

↓

)

28.19 (0.22

↓

)

23.98 (2.16

↓

)

16.02 (0.77

↓

)

26.54 (7.85

↓

)

18.36 (5.34

↓

)

14.52 (4.43

↓

)

10.34 (5.55

↓

)

FUTR

†

[25]

32.29 (4.71

↓

)

24.49 (3.33

↓

)

20.00 (2.46

↓

)

13.89 (2.87

↓

)

21.52 (11.83

↓

)

15.74 (9.44

↓

)

11.89 (8.25

↓

)

7.48 (8.04

↓

)

ActFusion (ours)

†

35.86

(3.69

↓

)

28.09

(0.51

↓

)

24.20

(0.59

↑

)

20.13

(0.23

↑

)

41.14

(1.66

↓

)

25.68

(1.43

↓

)

23.12

(0.36

↓

)

18.35

(3.71

↓

)

Breakfast

†

[20]

25.65 (0.23

↓

)

23.10 (0.32

↓

)

21.66 (0.76

↓

)

19.73 (1.82

↓

)

29.08 (0.59

↓

)

25.68 (1.7

↓

)

23.00 (2.59

↓

)

20.84 (4.36

↓

)

FUTR

†

[25]

27.85 (0.16

↑

)

24.68 (0.15

↑

)

22.91 (0.09

↑

)

20.98 (1.06

↓

)

32.36 (0.09

↑

)

28.96 (0.92

↓

)

25.49 (2.02

↓

)

23.56 (2.33

↓

)

ActFusion (ours)

†

27.90

(0.70

↓

)

24.69

(0.82

↓

)

22.99

(2.58

↓

)

22.42

(0.84

↓

)

33.47

(1.75

↓

)

30.28

(1.22

↓

)

30.69

(1.12

↑

)

26.77

(2.02

↓

)

6 Conclusion

We have presented ActFusion, a unified diffusion model that jointly addresses temporal action segmentation and long-term action anticipation in videos. The key to our method is anticipative masking, where learnable mask tokens replace unobserved frames, enabling simultaneous action segmentation of visible parts and anticipation of invisible parts. Our comprehensive experiments demonstrate that this unified approach not only outperforms existing task-specific models on both tasks but also reveals the mutual benefits of joint learning. Additionally, by evaluating our method both with and without ground-truth length information during LTA inference, we hope to motivate future research toward not using this information during testing. We believe that ActFusion demonstrates the potential of unifying complementary tasks in temporal action understanding, opening new directions for future research.

References

Y. Abu Farha and J. Gall. Uncertainty-aware anticipation of activities. In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops , pages 0-0, 2019. 2, 7

Y. Abu Farha, A. Richard, and J. Gall. When will you do what?-anticipating temporal occurrences of activities. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 5343-5352, 2018. 1, 2, 3, 7, 9

H. Ahn and D. Lee. Refining action segmentation with hierarchical video representations. In Proc. IEEE International Conference on Computer Vision (ICCV) , pages 16302-16310, 2021. 1, 2, 6

N. Aziere and S. Todorovic. Multistage temporal convolution transformer for action segmentation. Image and Vision Computing , 128:104567, 2022. 6

E. Bahrami, G. Francesca, and J. Gall. How much temporal long-term context is needed for action segmentation? In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 10351-10361, 2023. 2, 6

H. Bao, L. Dong, S. Piao, and F. Wei. Beit: Bert pre-training of image transformers. arXiv preprint arXiv:2106.08254 , 2021. 2

D. Baranchuk, A. Voynov, I. Rubachev, V. Khrulkov, and A. Babenko. Label-efficient semantic segmentation with diffusion models. In International Conference on Learning Representations , 2021. 3

N. Behrmann, S. A. Golestaneh, Z. Kolter, J. Gall, and M. Noroozi. Unified fully and timestamp supervised temporal action segmentation via sequence to sequence translation. In Proc. European Conference on Computer Vision (ECCV) , pages 52-68. Springer, 2022. 1, 2, 6, 21

J. Carreira and A. Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 6299-6308, 2017. 7

L. Chen, M. Li, Y. Duan, J. Zhou, and J. Lu. Uncertainty-aware representation learning for action segmentation. In IJCAI , volume 2, page 6, 2022. 6

M.-H. Chen, B. Li, Y. Bao, and G. AlRegib. Action segmentation with mixed temporal domain adaptation. In Proc. IEEE Winter Conference on Applications of Computer Vision (WACV) , pages 605-614, 2020. 6

M.-H. Chen, B. Li, Y. Bao, G. AlRegib, and Z. Kira. Action segmentation with joint selfsupervised temporal domain adaptation. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 9454-9463, 2020. 1, 2, 6

S. Chen, P. Sun, Y. Song, and P. Luo. Diffusiondet: Diffusion model for object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 1983019843, 2023. 3

T. Chen, R. ZHANG, and G. Hinton. Analog bits: Generating discrete data using diffusion models with self-conditioning. In The Eleventh International Conference on Learning Representations , 2022. 3

J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT (1) , 2019. 2, 18, 19

G. Ding, F. Sener, and A. Yao. Temporal action segmentation: An analysis of modern techniques. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2023. 2

Z. Du and Q. Wang. Dilated transformer with feature aggregation module for action segmentation. Neural Processing Letters , pages 1-17, 2022. 6

W.-C. Fan, Y.-C. Chen, D. Chen, Y. Cheng, L. Yuan, and Y.-C. F. Wang. Frido: Feature pyramid diffusion for complex scene image synthesis. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 37, pages 579-587, 2023. 3

Y. A. Farha and J. Gall. Ms-tcn: Multi-stage temporal convolutional network for action segmentation. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 3575-3584, 2019. 1, 2, 6, 7, 17, 19

Y. A. Farha, Q. Ke, B. Schiele, and J. Gall. Long-Term Anticipation of Activities with Cycle Consistency. In Proc. German Conference on Pattern Recognition (GCPR) . Springer, 2020. 2, 3, 7, 9, 10, 17, 20

A. Fathi, X. Ren, and J. M. Rehg. Learning to recognize objects in egocentric activities. In CVPR 2011 , pages 3281-3288. IEEE, 2011. 6, 7, 20

C. Feichtenhofer, Y. Li, K. He, et al. Masked autoencoders as spatiotemporal learners. Advances in neural information processing systems , 35:35946-35958, 2022. 19, 21

S.-H. Gao, Q. Han, Z.-Y. Li, P. Peng, L. Wang, and M.-M. Cheng. Global2local: Efficient structure search for video action segmentation. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 16805-16814, 2021. 2, 6

D. Gong, J. Lee, D. Jung, S. Kwak, and M. Cho. Activity grammars for temporal action segmentation. Advances in Neural Information Processing Systems , 36, 2024. 2, 6

D. Gong, J. Lee, M. Kim, S. J. Ha, and M. Cho. Future transformer for long-term action anticipation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 3052-3061, 2022. 1, 2, 3, 7, 9, 10, 17, 20, 21

A. Gupta, J. Liu, L. Bo, A. K. Roy-Chowdhury, and T. Mei. A-act: Action anticipation through cycle transformations. arXiv preprint arXiv:2204.00942 , 2022. 2, 7

W. Harvey, S. Naderiparizi, V. Masrani, C. Weilbach, and F. Wood. Flexible diffusion modeling of long videos. Advances in Neural Information Processing Systems , 35:27953-27965, 2022. 3

K. He, X. Chen, S. Xie, Y. Li, P. Dollár, and R. Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 16000-16009, 2022. 2, 8, 19, 21

J. Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models. In Proceedings of the 34th International Conference on Neural Information Processing Systems , pages 6840-6851, 2020. 3, 7, 15

Y. Huang, Y. Sugano, and Y. Sato. Improving action segmentation via graph-based temporal reasoning. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 14024-14034, 2020. 2, 6

K. Ishihara, G. Nakano, and T. Inoshita. Mcfm: Mutual cross fusion module for intermediate fusion-based action segmentation. In 2022 IEEE International Conference on Image Processing (ICIP) , pages 1701-1705. IEEE, 2022. 6

Y. Ishikawa, S. Kasai, Y. Aoki, and H. Kataoka. Alleviating over-segmentation errors by detecting action boundaries. In Proc. IEEE Winter Conference on Applications of Computer Vision (WACV) , pages 2322-2331, 2021. 1, 2, 6

S. Karaman, L. Seidenari, and A. Del Bimbo. Fast saliency based pooling of fisher encoded dense trajectories. In ECCV THUMOS Workshop , volume 1, page 5, 2014. 2

Q. Ke, M. Fritz, and B. Schiele. Time-conditioned action anticipation in one shot. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 9925-9934, 2019. 1, 2, 3, 7

G.-h. Kim and E. Kim. Stacked encoder-decoder transformer with boundary smoothing for action segmentation. Electronics Letters , 58(25):972-974, 2022. 6

H. Kuehne, A. Arslan, and T. Serre. The language of actions: Recovering the syntax and semantics of goal-directed human activities. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 780-787, 2014. 2, 6, 7, 20

H. Kuehne, A. Richard, and J. Gall. Weakly supervised learning of actions from transcripts. Computer Vision and Image Understanding , 163:78-89, 2017. 2, 21

C. Lea, M. D. Flynn, R. Vidal, A. Reiter, and G. D. Hager. Temporal convolutional networks for action segmentation and detection. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 156-165, 2017. 1, 2

J. Li, P. Lei, and S. Todorovic. Weakly supervised energy-based learning for action segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , October 2019. 21

M. Li, L. Chen, Y. Duan, Z. Hu, J. Feng, J. Zhou, and J. Lu. Bridge-prompt: Towards ordinal action understanding in instructional videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 19880-19889, 2022. 6

S.-J. Li, Y. AbuFarha, Y. Liu, M.-M. Cheng, and J. Gall. Ms-tcn++: Multi-stage temporal convolutional network for action segmentation. IEEE transactions on pattern analysis and machine intelligence , 2020. 6

X. Li, W. Chu, Y. Wu, W. Yuan, F. Liu, Q. Zhang, F. Li, H. Feng, E. Ding, and J. Wang. Videogen: A reference-guided latent diffusion approach for high definition text-to-video generation. arXiv preprint arXiv:2309.00398 , 2023. 3

D. Liu, Q. Li, A.-D. Dinh, T. Jiang, M. Shah, and C. Xu. Diffusion action segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 1013910149, 2023. 1, 2, 3, 5, 6, 7, 17, 19, 20, 21

M. Nawhal, A. A. Jyothi, and G. Mori. Rethinking learning approaches for long-term action anticipation. In European Conference on Computer Vision , pages 558-576. Springer, 2022. 2, 3, 7

M. Nawhal and G. Mori. Activity graph transformer for temporal action localization. arXiv preprint arXiv:2101.08540 , 2021. 1

J. Park, D. Kim, S. Huh, and S. Jo. Maximization and restoration: Action segmentation through dilation passing and temporal reconstruction. Pattern Recognition , 129:108764, 2022. 6

A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems , 32, 2019. 21

C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research , 21(140):1-67, 2020. 17, 18

A. Richard, H. Kuehne, and J. Gall. Weakly supervised action learning with rnn based fineto-coarse modeling. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 754-763, 2017. 7, 21

M. Rohrbach, S. Amin, M. Andriluka, and B. Schiele. A database for fine grained activity detection of cooking activities. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 1194-1201. IEEE, 2012. 2

R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 10684-10695, 2022. 3

F. Sener, D. Singhania, and A. Yao. Temporal aggregate representations for long-range video understanding. In Proc. European Conference on Computer Vision (ECCV) , pages 154-171. Springer, 2020. 1, 2, 3, 6, 7, 17, 20

P. Shaw, J. Uszkoreit, and A. Vaswani. Self-attention with relative position representations. arXiv preprint arXiv:1803.02155 , 2018. 18

J. Song, C. Meng, and S. Ermon. Denoising diffusion implicit models. In International Conference on Learning Representations , 2020. 3, 4, 7, 15

Y. Song and S. Ermon. Generative modeling by estimating gradients of the data distribution. Advances in Neural Information Processing Systems , 32, 2019. 3

Y. Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, and B. Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations , 2020. 3

Y. Souri, Y. A. Farha, F. Despinoy, G. Francesca, and J. Gall. Fifa: Fast inference approximation for action segmentation. In Pattern Recognition: 43rd DAGM German Conference, DAGM GCPR 2021, Bonn, Germany, September 28-October 1, 2021, Proceedings , pages 282-296. Springer, 2022. 2

S. Stein and S. J. McKenna. Combining embedded accelerometers with computer vision for recognizing food preparation activities. In Proceedings of the 2013 ACM international joint conference on Pervasive and ubiquitous computing , pages 729-738, 2013. 6, 7, 20

Z. Tong, Y. Song, J. Wang, and L. Wang. Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training. Advances in neural information processing systems , 35:10078-10093, 2022. 2, 5

A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin. Attention is all you need. Proc. Neural Information Processing Systems (NeurIPS) , 30, 2017. 5

Z. Wang, Z. Gao, L. Wang, Z. Li, and G. Wu. Boundary-aware cascade networks for temporal action segmentation. In Proc. European Conference on Computer Vision (ECCV) , pages 34-51. Springer, 2020. 1, 6

C. Wei, K. Mangalam, P.-Y. Huang, Y. Li, H. Fan, H. Xu, H. Wang, C. Xie, A. Yuille, and C. Feichtenhofer. Diffusion models as masked autoencoders. arXiv preprint arXiv:2304.03283 , 2023. 3

Z. Xu, Y. S. Rawat, Y. Wong, M. Kankanhalli, and M. Shah. Don't pour cereal into coffee: Differentiable temporal logic for temporal action segmentation. In Advances in Neural Information Processing Systems (NeurIPS) . 1, 2, 6

R. Yang, P. Srivastava, and S. Mandt. Diffusion probabilistic modeling for video generation. Entropy , 25(10):1469, 2023. 3

F. Yi, H. Wen, and T. Jiang. Asformer: Transformer for action segmentation. In Proc. British Machine Vision Conference (BMVC) , 2021. 1, 2, 5, 6, 7, 17, 19

C. Zhang, C. Fu, S. Wang, N. Agarwal, K. Lee, C. Choi, and C. Sun. Object-centric video representation for long-term action anticipation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages 6751-6761, 2024. 3, 7

Z. Zhong, M. Martin, M. Voit, J. Gall, and J. Beyerer. A survey on deep learning techniques for action anticipation. arXiv preprint arXiv:2309.17257 , 2023. 7

Appendix

In this appendix, we offer detailed descriptions and additional results, which are omitted in the main paper due to the lack of space. We provide a detailed explanation of diffusion models in Sec. A, training and inference algorithms in Sec. B, masking types in Sec. C, encoder and decoder layers in Sec. D, additional experimental results in Sec. E, more information of datasets in Sec. F, more experimental details in Sec. G, more qualitative results in Sec. H, and discussions of limitations and broader impacts in Sec. I.

A Diffusion models

In this section, we give an in-depth explanation of the diffusion models [29, 54] described in Sec. 3. The forward process adds noise to a data distribution x 0 ∼ q ( x 0 ) , following the noise distribution q based on the Markov property. Specifically, the forward process is defined by adding noise at an arbitrary time step s with a variance γ ( s ) as follows:

where S represents the entire time step. We can directly obtain the noisy data distribution x s at time step s without iteratively applying q due to the Markov property:

where δ ( s ) := 1 -γ ( s ) , ¯ δ ( s ) := ∏ s i =1 δ ( i ) , and ϵ ∼ N (0 , I ) . Instead of using γ ( s ) , 1 -δ ( s ) is utilized [29]. The posterior q ( x s -1 | x s , x 0 ) conforms to a Gaussian distribution and is expressed in terms of mean ˜ m s ( x s , x 0 ) and variance ˜ γ ( s ) using Bayes Theorem:

With a sufficiently large S and an appropriate variance schedule γ ( s ) , the noisy data x S follows an isotropic Gaussian distribution. Consequently, if we know the reverse distribution q ( x s -1 | x s ) , we can sample x S ∼ N (0 , I ) and reverse the process to obtain a sample from q ( x 0 ) . However, since q ( x s -1 | x s ) relies on the entire data distribution, we employ a neural network to approximate q ( x s -1 | x s ) as follows:

where m θ and Σ θ represent the predicted mean and co-variance, respectively, derived from the neural network. Predicting ϵ or x 0 in Eq. 16 is found to be effective [29]. In this work, we choose to predict x 0 using the neural network.

where

Algorithm 1 ActFusion Training

B Algorithms

We provide training algorithms of ActFusion in Alg. 1 and inference algorithms for TAS and LTA in Alg. 2 and Alg. 3, respectively.

Algorithm 2 ActFusion TAS Inference

Algorithm 3 ActFusion LTA Inference

delta_cumprod(s) : cumulative product of δ i , , ∏ s i =1 δ i

C Masking types

"

!

"

"

"

#

"

$

"

%

take cup

pour coffee

pour milk

stir coffee

#

'

:

none

#

(

:

ant

#

)

:

random

#

*

:

boundary

#

+

:

relation

dilated conv

add & IN

FFN

add

x

𝑁 E

መ

𝐴 enc

𝑾 enc

𝑭′

Q

K

V

⨂

+

⨂

𝑬

𝑩

dilated conv

add & IN

FFN

add

x

𝑁 D

መ

𝐴 0

𝑾 dec

𝐀 𝑠

Q

K

V

⨂

+

⨂

𝒀

𝑩

𝑠

concat

: relative position bias

𝑩

⨁

: addition

⨂

: multiplication

(b) encoder and decoder layer

̸

During training, a masking type is randomly selected among five: no mask M N , anticipative mask M A , random mask M R , relation mask M S , and boundary mask M B , for each iteration as detailed in Sec. 4.4. During inference, the no mask M N and the anticipative mask M A are utilized following the evaluation protocol [19, 65, 43, 20, 52, 25].

Figure S1 illustrates five types of masking. In Sec. 5.3, we introduced two types of masking: anticipative masking and random masking. The anticipation mask M A and the random mask M R are defined for each masking, respectively. Additionally, we adopt the relation mask M S and the boundary mask M B introduced in DiffAct [43]. The relation mask M S is proposed to help the model learn relational dependencies between actions. It is defined by M S = ✶ ( A 0 i,a = 1) , i ∈ { 1 , 2 , ..., T } and a ∈ { 1 , 2 , ..., K } , with T representing the number of frames and K denoting the number of action classes. Here, a specific action class a is randomly selected for the mask. The boundary mask M B hides visual features near the boundaries based on the soft ground truth ¯ B to manage the uncertainty associated with action transitions. Specifically, it is defined by M B i = ✶ ( ¯ B < 0 . 5) , where i ∈ { 1 , 2 , ..., T } .

D Model architecture

We employ a modified version of ASFormer [65] utilized in DiffAct [43] as the baseline model. An encoder g consists of N E number of layers and a decoder h consists of N D number of layers as described in Sec.4.2. Figure S2 illustrates detailed operations of the encoder and the decoder layers. An encoder layer consists of dilated convolution followed by dilated attention, instance normalization, and a feed-forward network. Additionally, we add relative position bias B to attention scores to consider relative position relations among actions [48]. Similarly, a decoder layer comprises dilated convolution followed by dilated attention, instance normalization, and a feed-forward network. The output embedding E from the encoder is concatenated to the input of the decoder after dilated convolutions. Subsequently, they are embedded as queries and keys in the dilated attention operation. We refer the reader to [65, 43] for more details.

(a) Results on TAS

(b) Results on LTA

E Additional experiments

We provide additional experimental results, maintaining the same experimental settings as described in Sec.5.2 unless otherwise specified. The evaluation is conducted on the 50 Salads dataset.

Conditioning features. We investigate different types of conditioning features: masked features F ′ , output embeddings E from the encoder, and the encoder prediction ˆ A . Table S1 shows overall results, where using E is more effective than using other features. The effectiveness of utilizing the encoder in our approach becomes apparent when comparing the first and second rows, where a significant performance drop is observed in the absence of the encoder. Since mask tokens replace visual tokens before going into the encoder, it is crucial for them to learn action relations through the encoder. Comparing the second and third rows, we also observe that utilizing features from intermediate layers yields slightly better results than using encoder predictions for both TAS and LTA.

Position embedding. In Table S2, we explore different types of position embeddings: relative position bias [48], relative position embedding [53], and absolute position embedding [15]. Comparing the first and second rows, we find that employing relative position bias enhances the overall performance in TAS and LTA. From the second and third rows, we observe that relative position bias is also more effective than using learnable relative position embedding. Using absolute position embedding leads to decreased performance in both TAS and LTA. We find that learnable embeddings often cause overfitting problems during training. As a result, we adopt relative position bias in our model.

(b) Results on LTA

(a) Results on TAS

Loss ablations. We conduct ablation studies on the loss functions: boundary loss, smoothing loss, and cross-entropy loss. Table R4 presents the results, demonstrating that the combination of bounding loss and smoothing loss is effective for both TAS and LTA. While the effectiveness of these losses in TAS is well-documented in previous research [19, 43, 65], their impact on LTA has been less explored. Notably, the smoothing loss leads to significant performance gains in both tasks, indicating that smoothed predictions are beneficial.

Effects of reconstruction loss L recon Masked auto-encoding is a technique used in training NLP models like BERT [15] and has recently been adapted to vision models [22, 28]. Inspired by this approach, we train our model to reconstruct input video features from the masked tokens as an auxiliary task. Specifically, we employ MLP layers on the encoder embeddings to reconstruct the input video features and apply mean squared error (MSE) loss L recon during training.

Table S4 shows the overall results on both TAS and LTA tasks. In TAS, overall performance increases. We conjecture that reconstruction helps the model gain a deeper understanding of the underlying data structure and temporal dynamics by predicting the missing parts of the input. In LTA, we find that reconstruction is more effective on relatively short-term anticipation. Since short-term predictions are often based on more immediate context, there is less uncertainty. As a result, reconstructing masked features helps the model capture immediate patterns and transitions more accurately. However, for long-term predictions, as the model attempts to predict further into the future, the uncertainty increases significantly. Long-term predictions involve more variables and potential changes, making them inherently less predictable. This increased uncertainty might cause performance degradation, making reconstruction less effective for action anticipation.

F Datasets

The 50 Salads [58] dataset consists of 50 videos depicting 25 individuals preparing a salad. With over 4 hours of RGB-D video data, the annotations include 17 fine-grained action labels and 3 high-level activities. Notably, 50 Salads videos are longer than those in the Breakfast dataset, averaging 20 actions per video. The dataset is partitioned into 5 splits for cross-validations, and we report the average performance across all splits. The Breakfast dataset is under the license of Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.

The Breakfast [36] dataset consists of 1,712 videos featuring 52 individuals preparing breakfast in 18 different kitchens. Each video is assigned to one of the 10 activities associated with breakfast preparation, utilizing 48 fine-grained action labels to define these activities. The average video duration is approximately 2.3 minutes, encompassing around 6 actions per video. The dataset is divided into 4 splits for cross-validations, and we report the average performance across all splits. The Breakfast dataset is under the license of Creative Commons Attribution 4.0 International License.

The GTEA dataset [21] comprises 28 videos capturing 11 action classes related to cooking activities. Each video includes 20 actions on average, and an average video duration is about a minute. The dataset is divided into 4 splits for cross-validations, and we report average performance across all splits. The GTEA dataset is under the license of Creative Commons Attribution 4.0 International License.

G Experimental details

Cross-task generalization. In Fig.1-(c), we conduct experiments on cross-task evaluations with DiffAct [43], FUTR [25], and TempAgg [52]. To evaluate DiffAct on LTA, we limit the input video frames to F 1: αT , with zero masks appended to future frame lengths concatenated to the encoder output embeddings, adhering to DiffAct's masking strategies. Subsequently, the decoder predicts future actions based on the observed video frames. To evaluate FUTR on TAS, we utilize the encoder of FUTR as an action segmentation model. Note that we report the performance of TempAgg on both TAS and LTA as provided in the original paper [52].

Evaluation without using ground-truth prediction length on LTA. Baseline models of Cycle Cons. [20] and FUTR [25] anticipate future actions and their corresponding durations. The predicted durations are then directly multiplied by the ground-truth prediction length, βT , to generate the final predictions. In Table 6, we also experimented with using a modified prediction length, rαT , instead

of the ground-truth length. Please note that we use a reproduced model for Cycle Cons. and the official model checkpoints for FUTR 2 .

Implementation details. We provide additional implementation details to complement those described in Sec. 5.2. Table S5 presents the specific hyperparameters used in our experiments for each dataset. All experiments are conducted on a single NVIDIA RTX-3080 GPU. We implement ActFusion using Pytorch [47] and some of the official code repository of DiffAct [43] 3 licensed under an MIT License.

H Qualitative results

We provide additional qualitative results for both successful and failure cases in Fig. S3 and Fig. S4, respectively. Figure S3 demonstrates the promising results on both TAS and LTA, showing the efficacy of joint learning these two tasks. In the failure cases, figure S4a highlights the importance of accurate segmentation on anticipation, where inaccurate action segmentation of the observed frames leads to wrong future anticipation. In Fig. S4b, ActFusion fails to detect the 'take knife' action class in action segmentation. However, the model anticipates missing action classes that are not observed in the observed frames. This implies that the model can infer missing actions based on the action relations of the observed and unobserved actions.

I Discussion

In this paper, we have proposed a unified diffusion model for action segmentation and anticipation, dubbed ActFusion. We have demonstrated the effectiveness of the proposed method through comprehensive experimental results, showing the bi-directional benefits of joint learning the two tasks. In this section, we will discuss the limitations, future work, and broader impact of the proposed method.

Limitations and future work. The proposed method shows the state-of-the-art results on TAS across three benchmark datasets. However, the frame-wise accuracy is slightly below compared to DiffAct [43]. We hypothesize that this discrepancy arises from the different masking strategies employed. In DiffAct, masking is applied to the output embeddings of the encoder, ensuring that the encoder always fully observes the visual features from the entire video. However, in our approach, masking is applied before the encoders, allowing for handling both visible and invisible tokens to unify the two tasks effectively. Consequently, the encoder may not always observe the entire visual features, potentially leading to slightly lower accuracy. This issue could potentially be addressed by employing reconstruction methods for masked features, similar to masked autoencoders [28, 22]. By further training the model to reconstruct the original features from the masked features, the encoder could be empowered to handle the masked features better. We leave this as our future work.

In future work, additional activity information could be integrated, particularly focusing on segmentwise action relations [8]. Moreover, exploring weakly-supervised learning approaches [39, 49, 37] could be beneficial for further enhancing the capabilities of our method.

Broader impact. To the best of our knowledge, we are the first to integrate action segmentation and anticipation within a unified model. We believe that our work lays the foundation for integrating these two tasks, offering substantial potential for real-world applications such as robots.

SUP

SUP

SUP

Split1, P10_webcam01_P10_milk

GT

TAS

SIL

spoon powder

stir milk

activity: milk

pour milk

LTA

observation

anticipation

(a)

GT

TAS

LTA

SIL

pour cereal

pour milk

activity: cereals

take bowl

P29_cam02_P29_cereals

observation

anticipation

(b)

GT

TAS

SIL

squeeze orange

pour juice

activity: juice

cut orange

LTA

P42_stereo01_P42_juice

take glass

observation

anticipation

(c)

SUP, failure

P33_stereo01_P33_tea activity: tea

SUP, failure

(b)

GT

TAS

SIL

take cup

add teabag

pour water

LTA

pour milk

pour coffee

observation

anticipation

(a)

GT

TAS

activity: sandwich

LTA

observation

anticipation

SIL

take knife

cut bun

smear butter

put toppingontop

['SIL', 'take_knife', 'cut_bun', 'smear_butter', 'put_toppingOnTop', 'put_bunTogether', 'SIL']

['SIL', 'cut_bun', 'take_knife', 'smear_butter', 'put_toppingOnTop', 'SIL']

NeurIPS Paper Checklist

The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.

Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:

You should answer [Yes] , [No] , or [NA] .

[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.

Please provide a short (1-2 sentence) justification right after your answer (even for NA).

The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include it (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.

The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation. While "[Yes] " is generally preferable to "[No] ", it is perfectly acceptable to answer "[No] " provided a proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] " or "[NA] " is not grounds for rejection. While the questions are phrased in a binary way, we acknowledge that the true answer is often more nuanced, so please just use your best judgment and write a justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.

IMPORTANT, please:

Delete this instruction block, but keep the section heading 'NeurIPS paper checklist" ,

Keep the checklist subsection headings, questions/answers and guidelines below.

Do not modify the questions and only use the provided macros for your answers .

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Our main claims presented in the abstract and introduction accurately reflect the paper's contributions and scope.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: See Sec. I.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We provide theoretical explanation of diffusion models on Sec. 3 and A. Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We provide experimental details in Sec. 5 and Sec. G.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We include the code and instructions for reproduction in the supplementary. The training and validation data is available online.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We specify all the training details in the main and supplementary.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: We do not report error bars due to the limited computational resources.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We include the amount and type of resources used for training in Sec. G.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We follow the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We provide the broader impact in Sec. I.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: We do not release data or models that can be highly misused.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We cite all the papers that are relevant to the code, models, and datasets. We include the license of each asset in Sec. F and G.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: We include trained models in the supplementary for reproduction of our main results.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: We did not croudsource any information for this research.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: We did not croudsource any information for this research.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper extends DiffAct to perform both action segmentation and action anticipation. An anticipative masking with a learnable mask token is proposed. Experiments are conducted on the three common benchmark datasets.\n\nSTRENGTHS:\n1. The motivation for unifying action segmentation and action anticipation is reasonable, given the task similarity. It is also intuitive and reasonable to extend a generative framework from segmentation to anticipation, given the generative nature of anticipation.\n\n2. The learnable mask token is interesting.\n\n3. The ablation studies are relatively comprehensive.\n\n4. Codes are provided in the supplementary.\n\nWEAKNESSES:\n1. The major technical problem is that the proposed method assumes the ground truth video length is known for action anticipation at the test time. Without the ground truth video length T, the anticipative mask M^A can not be constructed during the inference time for anticipation. This is also shown in the code provided, where 'full_len' is input into the ddim inference function. This is problematic, conflicting with the goal of anticipation, and leading to incomparable experimental results.\n\n2. The technical novelty is limited. The main contribution is extending the DiffAct method with an anticipative mask, while other modules are from existing methods. But given the contribution of unifying the segmentation and anticipation tasks, this is not a deciding factor for me.\n\n3. It will be better to conduct experiments on Assembly101. Given the small data size and the saturated performance on GTEA, I would recommend a transition from GTEA to Assembly101 for this task.\n\nMinor:\n- In Figure 1c, what do the triangle and the circle mean?\n- In Table 1, some recent methods are missing, such as MVGA ICCV23 and RTK ICCV23.\n\nQUESTIONS:\nNA\n\nLIMITATIONS:\nThe limitations were briefly discussed.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 1, "SOUNDNESS_CORRECTNESS": 1, "DATE": "2024-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new unified diffusion model called ActFusion, which solves the tasks of Time Action Segmentation (TAS) and Long Term Action Prediction (LTA) in a joint learning framework. To unify the two tasks, the model effectively handles the visible and invisible parts of the sequence during the training phase; The visible part is used for observed video frames, while the invisible part is used for future expectations. The experiment showed that the model achieved state-of-the-art performance in standard benchmark tests of 50 Salad, Breakfast, and GTEA.\n\nSTRENGTHS:\n1. The design of the ActFusion model is novel, with its anticipative masking strategy and random masking method unifying the tasks of TAS and LTA.\n2. The model enhances its performance on both tasks through mutual promotion of TAS and LTA.\n3. The reported results in the paper surpass existing techniques on multiple evaluation metrics, demonstrating significant performance improvements.\n\nWEAKNESSES:\n1. What is the difference between the Diffusion model used in the paper and the DiffAct model? They appear to be similar overall, despite using different Masks to unify the tasks of TAS and LTA.\n2. The paper employs so many loss functions for model training; the authors should analyze the impact of different losses rather than simply exploring the effects of the encoder and decoder.\n3. Since the denoising process of the diffusion model is a time-consuming task, I am interested in the computational efficiency of the model proposed in the paper. The authors are advised to provide information on the model's GFLOPs, parameter scale, and inference time.\n4. The authors need to further improve the interpretation of the figures, such as what the circles and triangles in Figure 1(c) represent.\n\nQUESTIONS:\n1. What is the difference between the Diffusion model used in the paper and the DiffAct model? They appear to be similar overall, despite using different Masks to unify the tasks of TAS and LTA.\n2. The paper employs so many loss functions for model training; the authors should analyze the impact of different losses rather than simply exploring the effects of the encoder and decoder.\n3. Since the denoising process of the diffusion model is a time-consuming task, I am interested in the computational efficiency of the model proposed in the paper. The authors are advised to provide information on the model's GFLOPs, parameter scale, and inference time.\n4. The authors need to further improve the interpretation of the figures, such as what the circles and triangles in Figure 1(c) represent.\n\nLIMITATIONS:\nIn summary, the authors have creatively used different Masks to unify the tasks of TAS and LTA with a single model and achieved significant results. However, compared to existing models, there does not seem to be much modification, highlighting a disadvantage in innovation.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe author introduce a unified diffusion model for temporal action segmentation (TAS) and long-term action anticipation (LTA), dubbed ActFusion, where a single model is jointly trained to address these two problems effectively. A new anticipative masking is presented for the effective unification of two tasks, along with random masking to learn intra-action relations. ActFusion achieves the state-of-the-art performance on both TAS and LTA, demonstrating the effectiveness of joint learning of two tasks across standard benchmark datasets, 50 Salads and Breakfast, and GTEA.\n\nSTRENGTHS:\n* Originality\n    * The paper presents a approach to integrating two popular vision tasks, temporal action segmentation and long-term action anticipation, within a unified model. This is the first time these two problems have been investigated together in a single framework, highlighting the originality of the research. \n* Clarity\n    * The paper is clearly articulated and provides sufficient details. It offers an adequate background on diffusion model and how it is utilized. The inclusion of pseudo-code and actual code facilitates a better understanding for the reader. Additionally, the experimental settings are thoroughly described, which aids in the reproducibility of the results.\n\nWEAKNESSES:\n* The methodology lacks novelty. The model architecture, loss function (Cross-Entropy Loss, Temporal Smoothness Loss, Boundary Alignment Loss) and mask strategy (no mask, relation mask, boundary mask) closely resemble those used in the Diffusion Action Segmentation (https://arxiv.org/pdf/2303.17959v2). The primary distinction in this work is only its extension to include the long-term action anticipation task, and the introduction of anticipative masking.\n* The performance improvement relative to other state-of-the-art works is marginal. On TAS task, when compared to DiffAct, this model shows an approximate 1-point improvement across various metrics (F1, edit score, frame-wise accuracy) on different benchmarks (50 Salads, Breakfast, GTEA). \n* The model consistently achieves sub-optimal results when assessed using frame-wise accuracy as the metric on TAS task. As the authors point out, this could potentially be addressed by employing reconstruction methods for masked features. I am eager to see how these adjustments could enhance the model's performance.\n* More ablation study for LTA task could be included, e.g., How important is past context for the models (\\alpha) ? How far into the future can models predict (\\beta)? Table 2 reports only a limited range of settings. A more thorough analysis on these aspects would be highly valuable.\n\nQUESTIONS:\n* The approach is currently limited to predicting action labels within a closed set. To extend this work to predict open-set action labels, what can be done?\n* While \"segmentation helps anticipation\" is evident from table 3, \"anticipation helps segmentation\" is considerably less significant in table 4. What's the reason behind this?\n* If further training the model to reconstruct the original features from the masked features, would it also improve the LTA task?\n* Paper writing\n    * What does circle and triangle means in figure 1(c)?\n    * In table2, why there are multiple underlined values (suppose to be second-highest value) in each column?\n\nLIMITATIONS:\n* Limitation\n    * The author notes the sub-optimal performance when using frame-wise accuracy to evaluate the TAS task and suggests a potential solution, though its effectiveness remains unproven. I am eager to see how these proposed adjustments might improve the model's performance.\n* Potential negative societal impact\n    * There is no potential negative societal impact of this work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-24"}