ActFusion: a Unified Diffusion Model for Action Segmentation and Anticipation

Authors: dayoung  gong
suha  kwak
minsu  cho
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-12 07:41:57

Abstract: Temporal action segmentation and long-term action anticipation are two popular vision tasks for the temporal analysis of actions in videos. 
Despite apparent relevance and potential complementarity, these two problems have been investigated as separate and distinct tasks. In this work, we tackle these two problems, action segmentation, and action anticipation, jointly using a unified diffusion model dubbed ActFusion. 
The key idea to unification is to train the model to effectively handle both visible and invisible parts of the sequence in an integrated manner;
the visible part is for temporal segmentation, and the invisible part is for future anticipation. 
To this end, we introduce a new anticipative masking strategy during training in which a late part of the video frames is masked as invisible, and learnable tokens replace these frames to learn to predict the invisible future.
Experimental results demonstrate the bi-directional benefits between action segmentation and anticipation.
ActFusion achieves the state-of-the-art performance across the standard benchmarks of 50 Salads, Breakfast, and GTEA, outperforming task-specific models in both of the two tasks with a single unified model through joint learning.

Main paper: 
 introduction: 
 In everyday life, when interacting with people, we anticipate their future actions while recognizing their actions observed in the present and the past. Similarly, for effective human-robot interaction, robotic agents must recognize ongoing actions while anticipating future behaviors. Two essential tasks in computer vision for such a temporal understanding of human actions are temporal action segmentation (TAS) [38, 19, 32, 61, 12, 65, 3, 43, 63, 8] and long-term action anticipation (LTA) [2, 34, 25, 45] . The task of TAS aims at translating observed video frames into a sequence of action segments, while the goal of LTA is to predict a plausible sequence of actions in the future based on the observed video frames. These tasks are closely related in terms of understanding the relations between actions; recognizing actions in the present and the past may improve anticipating action in the future, and the ability to anticipate the future may also enhance recognizing observable actions when facing visual ambiguities.Despite the apparent relevance and potential complementarity, these two problems have been investigated as separate and distinct tasks. While a growing body of work has shown remarkable progress on both TAS and LTA, these methods are primarily designed for one task (see Fig. 1a ) and show poor generalization when applied to the other (e.g., see FUTR [25] and DiffAct [43] in Fig. 1c ). Some prior methods tackle both tasks simultaneously, but they rely on task-specific architectures and require separate training for each individual task (e.g., see TempAgg [52] in Fig. 1c ).In this paper, we introduce ActFusion, a unified diffusion model that addresses TAS and LTA through a single architecture and training process, as illustrated in Fig. 1b . The core idea of our Tasks-specific models such as DiffAct [43] for TAS and FUTR [25] for LTA exhibits poor performance on cross-task evaluations. ActFusion outperforms task-specific models on both TAS and LTA, including TempAgg [52] , which trains separate models for each task. Note that the performance of ActFusion is evaluation result of a single model through a single training process. The reported performance represents the average of each task in the original paper or evaluated with the official checkpoint (See Sec. F for details).unified approach is to train the model to effectively handle two different parts of the sequence: visible parts for action segmentation and invisible parts for action anticipation. Accordingly, we propose a new anticipative masking strategy in which a late part of the video frames is masked as invisible, and learnable tokens replace it. We also introduce a random masking strategy to help the model accurately identify actions when some parts of the video are ambiguous [59] , drawing inspiration from previous methods [15, 6, 28, 59] . ActFusion generates action sequences from Gaussian noise through iterative denoising, conditioned on both visual features and mask tokens. Through this process, the model jointly learns to classify action labels of visual features and predict future actions of mask tokens. As shown in Fig. 1c , ActFusion achieves superior performance in both action segmentation and anticipation, demonstrating stronger cross-task generalization compared to previous methods [43, 25, 52] .Furthermore, we find that the benchmark evaluation of some prior LTA methods [2, 20, 25] has exploited the ground-truth length of input videos in testing time, which makes the evaluation unrealistic and problematic since the duration of future actions is supposed to be unknown in advance. To address this issue, we conduct comprehensive experiments both with and without ground-truth length information, providing insights into more realistic deployment scenarios.Our contribution can be summarized as follows: 1) We introduce ActFusion, a unified diffusion model that jointly addresses TAS and LTA through a single training process. 2) We present anticipative masking for effective task unification, along with random masking to enhance robustness against visual ambiguities. 3) Comprehensive experimental results show that jointly learning both tasks provides bi-directional benefits. ActFusion achieves state-of-the-art performance on TAS and LTA across benchmark datasets -50 Salads and Breakfast, and GTEA -demonstrating the effectiveness of the proposed method. 
related work: 
 Temporal action segmentation (TAS). The goal of TAS [19, 65, 43, 63, 12, 23, 5, 8, 32, 3, 16] is to classify frame-wise action labels in a video. Earlier approaches employ temporal sliding windows [50, 33] for action segment detection, grammar-based methods [37, 36] have been introduced to incorporate a temporal hierarchy of actions during segmentation. Temporal convolution networks [38, 19] and Transformer-based models [65, 8] are introduced based on deep learning methods. Since learning long-term relations of actions from activity videos is challenging, a series of work has been proposed to develop refinement strategies [30, 32, 23, 12, 57, 63, 3, 24] that can be applied to the TAS models [19, 65] . Recently, DiffAct [43] is proposed to iteratively denoising action predictions conditioned on the input video features adopting the diffusion process. In a similar spirit, the proposed ActFusion is based on the diffusion process, focusing on unifying TAS and LTA through anticipative masking. We demonstrate the bidirectional benefits between the two tasks, showing that learning segmentation along with anticipation is effective.Long-term action anticipation (LTA). LTA [2, 1, 34, 20, 52, 25, 44, 26] has recently emerged as a crucial task for predicting a sequence of future actions in long-term videos. Initial models use RNNs and CNNs [2] , while time-conditioned anticipation [34] introduces one-shot anticipation of specific future timestamps. A GRU-based model [20] used cycle-consistent relations of past and future actions. Sener et al. [52] proposed TempAgg, a multi-granular temporal aggregation method for action anticipation and recognition, utilizing different model architectures and task-specific losses for the two tasks. Gong et al. [25] propose a transformer model for parallel anticipation, dubbed FUTR, empirically showing that learning action segmentation as an auxiliary task is helpful for anticipation. Nawhal et al. [44] propose a two-stage learning approach for LTA and Zhang et al. [66] present object-centric representations using visual-language pre-trained models for LTA. While previous work [52, 20, 25, 44] adopts TAS as an auxiliary task to help learn LTA, a unified model evaluated on both tasks is rarely explored, often showing poor cross-task generalization performance (Fig. 1c ). The exception is TempAgg, which requires separate training and model architecture for the two tasks.In this work, we present a unified model that tackles both TAS and LTA in a single training process.Diffusion models. Recent success in denoising diffusion models [55, 29, 56] opens a new era of computer vision research. The diffusion models learn the original data distribution through the iterative denoising process. Diffusion models have recently shown notable success in various domains, such as image generation [51, 14] , video generation [42, 27, 64] , object detection [13] , semantic segmentation [7] , temporal action segmentation [43] , self-supervised learning [62] , and etc [18] . Recently, DiffMAE [62] integrates masked autoencoders with the diffusion models, where the model learns to denoise masked input while learning data distributions through generative pre-training of visual representations. In this work, we present a unified diffusion model that effectively integrates TAS and LTA through masking, where the model learns to denoise action labels conditioned on both visual and mask tokens. In this way, the model can effectively learn temporal relations between actions by classifying visual tokens and inferring missing actions of the mask tokens. 
preliminary: 
 In this section, we give a brief overview of diffusion models [29, 54] . The diffusion models learn a data distribution by mapping and denoising noises from the original data distribution. The training process of diffusion models involves forward and reverse processes from random noise. During the forward process, Gaussian noise is added to the original data, while in the reverse process, a neural network learns to reconstruct the original data by iteratively removing noise.The forward process, also known as the diffusion process, transforms the original data x 0 into noisy data x s :EQUATIONHere, a noise ϵ ∼ N (0, I) is added to the original data distribution x 0 following the decreasing function γ(s) of time step s ∈ {1, 2, ..., S}, where S represents the entire forward time-step. Note that the function γ(s) determines the intensity of the noise added to the original data following the pre-defined variance schedule [29] .In the reverse process, a neural network f (x s , s) is trained to recover the original data x 0 from noisy data x s using l 2 regression loss:EQUATION)During inference, the model f iteratively denoises pure noise x S to reconstruct the original data x 0 , i.e., x S → x S-∆ → ... → x 0 , following the updating rule [29, 54] . We refer the reader to [29, 54] and Sec. A for more details.In our context, the neural network learns to generate action sequences from Gaussian noise, conditioned on visible features for action segmentation and mask tokens for action anticipation. : visual tokens• Training: all masks • Inference: none (TAS), ant (LTA) During training, we randomly select one of five masking strategies and apply it to input video frames F , replacing masked regions with learnable tokens to obtain masked features F ′ . These features are processed by the encoder g to produce visual embeddings E, which condition the decoder h to denoise action labels from A s to Â0 at time-step s. For inference, we use different masking strategies depending on the task: no masking for TAS and anticipative masking for LTA. The decoder then iteratively denoises action labels following ÂS → ÂS-∆ → ... → Â0 using the DDIM update rule [54] .dec ℎ # 𝑨 "#∆ # 𝑨 " 0 𝑇 iterative denoising [M] [M] [M] [M] [M] [M] [M] [M] [M] [M] [M] [M] 
proposed approach: 
 In this section, we present ActFusion, our unified diffusion model for action segmentation and anticipation. We first describe the problem setup in Sec. 4.1, followed by the model architecture in Sec. 4.2. We then introduce our masking strategies in Sec. 4.3 and training objectives in Sec. 4.4.Temporal action segmentation (TAS) aims to classify input video frames into a sequence of predefined action classes, while long-term action anticipation (LTA) predicts future actions based on partially observed video sequences. Formally, given a video sequenceF = [F 1 , F 2 , • • • , F T ] of length T , TAS predicts frame-wise action labels A = [A 1 , A 2 , • • • , A T ],where each A i is a one-hot vector representing the action class. In LTA, given the first N O = ⌈αT ⌉ observed frames, the goal is to anticipate action labels for the subsequent N A = ⌈βT ⌉ frames, where α ∈ [0, 1] and β ∈ [0, 1 -α] represent the observation and anticipation ratios, respectively. Here, ⌈•⌉ denotes the ceiling function.ActFusion aims to unify TAS and LTA by leveraging an encoder-decoder architecture with adaptive masking strategies. Figure 2 illustrates the overall pipeline, where ActFusion consists of a masked encoder g and a denoising decoder h. The encoder obtains visual features and mask tokens as input and generates embedded tokens as output. The decoder then progressively reduces noises through an iterative denoising process conditioned on the embedded tokens.During training, we randomly sample a time step s ∈ {1, 2, ..., S} for each iteration and add noise ϵ ∼ N (0, I) to the ground-truth action labels A 0 following Eq. 1, resulting in noisy action labels A s . The decoder then aims to denoise A s to reconstruct the original action labels Â0 . During inference, the decoder starts with Gaussian noise ÂS and progressively denoises it following the DDIM [54] update rule to generate final predictions Â0 , i.e., ÂS → ÂS-∆ → ... → Â0 .The key to our unified approach lies in training the model to effectively process both visible and invisible parts of the sequence, where the visible part corresponds to observed video frames and the invisible part represents future frames to be anticipated. To this end, we introduce anticipative masking that replaces unobserved frames with learnable mask tokens, enabling the model to learn future predictions. We apply this masking strategy consistently during both training and inference to achieve joint learning of action segmentation and anticipation. We further incorporate random masking, where video frames are randomly masked to enhance robustness against visual ambiguities. Both masking strategies are detailed in Sec. 4.3.Input structuring. Given input video features F ∈ R T ×C with T frames of C feature dimensions, we start by defining a binary mask matrix M ∈ {0, 1} T ×1 . This mask serves as a frame selector, where a value of 0 indicates a frame to be masked and replaced by mask tokens, while a value of 1 indicates an unmasked frame. The learnable mask token, denoted as m ∈ R 1×C , replaces the visual features in frames selected for masking, producing the input features F ′ for the encoder:EQUATIONwhere ⊙ denotes element-wise multiplication, and 1 i×j represents a matrix of ones with dimensions i × j. Here, M is broadcasted along the channel dimension and m is broadcasted along the temporal dimension to match the size of F .For our model architecture, we use a modified version of ASFormer [65] as used in DiffAct [43] , which employs dilated attention to capture both local and global relations in the input sequence (See Fig. S2 for the detailed model architecture).Encoder. The encoder consists of the N E layers, each consisting of a dilated 1-d convolution followed by instance normalization, dilated attention, and a feed-forward network [65] . In the dilated attention mechanism, the receptive field is limited to a local window size w = 2 i for the i-th layer, where the increasing dilation captures progressively broader temporal relations. The output of each layer is combined with its input via a residual connection before proceeding to the subsequent layer.Given the input features F ′ , the encoder g produces embedded tokens E ∈ R T ×D as:EQUATIONwhere D represents the dimensions of embedded tokens. A fully-connected (FC) layer W enc ∈ R D×K is applied to E to obtain frame-wise classification logits from the encoder followed by a softmax:EQUATIONwhere K is the number of action classes and σ is the softmax.Decoder. The decoder is composed of N D sequential layers. Each layer consists of dilated 1-d convolution, dilated attention, instance normalization, and feed-forward networks. As in the encoder, the dilation ratio for the i-th layer is set to w = 2 i . The output of each layer is combined with its input through a residual connection before being passed into the subsequent layer. Given a time step s, noisy action labels A s at time step s, and encoder embeddings E, the decoder h produces output embeddings Y according to:EQUATIONThe final action label predictions Â0 are obtained by projecting Y through a fully-connected layer W dec ∈ R D×K followed by softmax:EQUATIONWe introduce two distinct masking strategies: anticipative masking and random masking.Anticipative masking. Anticipative masking enables joint learning of TAS and LTA by separating visible and invisible parts of the input sequence. This strategy masks the latter portion of video frames, requiring the model to predict future actions based on observed frames. Given a video length T , we define a binary anticipation mask M A ∈ {0, 1} T that sets visible frames to 1 and invisible frames to 0:M A i = 1 i ≤ N O, where 1 is an indicator function and N o represents the number of observed frames. Unlike causal masking [60] that prevents attending to future tokens, our anticipative masking allows interactions among visible tokens while maintaining a clear boundary for anticipation.Random masking. Random masking aims to provide robustness in prediction when parts of video frames are missing or ambiguous [59] . A video is first divided into pre-defined clips of size Q, resulting in N P = ⌈ T Q ⌉ total clips. Then, N R clips are randomly selected to be masked. A binary random mask is defined byM R i = 1(∃j ∈ P, (j -1)Q < i ≤ jQ), where P is a randomly selected subset of {1, • • • , N P } with |P| = N R .For fully observable scenarios, we adopt a no mask strategy M N where all frames remain visible. Additionally, we adopt two masking strategies [43] , the relation mask M S and the boundary mask M B , as illustrated in Fig. 2 . The relation mask M S randomly masks segments associated with an action class to learn inter-action dependencies. The boundary mask M B masks frames at action transitions to enhance boundary detection. See Sec. C for the detailed formulations of the masks.During training, one of five masking strategies is randomly selected: no mask M N , anticipative mask M A , random mask M R , relation mask M S , and boundary mask M B . For inference, no mask M N is used for TAS and anticipative mask M A for LTA.The model is trained with three types of losses: cross-entropy loss L ce for frame-wise classification, boundary smoothing loss L smo [19] , and boundary alignment loss L bd [43] . These losses are applied to both encoder and decoder, where the encoder serves as an auxiliary task to enhance discrimination ability. Given the ground-truth action label A 0 ∈ R T ×K and the predictions Â ∈ R T ×K , the cross entropy loss is defined by:EQUATIONTo prevent over-segmentation errors, a temporal smooth loss [19] between adjacent frames based on a truncated mean squared error over the frame-wise log probabilities are defined by:EQUATIONwhere the difference of the log probabilities of two adjacent frames is truncated with a threshold value. For precise boundary detection, the boundary alignment loss [43] is employed based on the binary cross-entropy loss: where B = κ(B) represents a softened version of the ground-truth action boundaryEQUATIONB i = 1(A 0 i ̸ = A 0 i+1), achieved through a Gaussian kernel κ. The encoder and decoder losses are defined byEQUATIONEQUATIONwhere λ denotes a scaling factor. The total loss is defined asL total = L enc + L dec . 
experiments: 
 In this section, we conduct experiments to demonstrate the effectiveness of our model. All reported experimental results are obtained from inference using a single unified model. We evaluate our method on three widely-used benchmark datasets: 50 Salads [58] , Breakfast [36] , and GTEA [21] (see Sec. F for details). All three datasets are used to evaluate on TAS, while 50 Salads and Breakfast are used for evaluating LTA, following the protocols of the previous work [19, 65, 43, 20, 52, 25] .Evaluation metrics. For evaluation metrics for TAS, we report F1@{10, 25, 50} scores, the edit score, and frame-wise accuracy [19, 65, 43] . The F1 scores and the edit score are segment-wise metrics, and accuracy is a frame-wise metric. Mean over classes accuracy (MoC) is adopted as an evaluation metric for LTA [2, 34, 52, 25] .Implementation details For input video features, we utilize the pre-trained I3D features [9] for all datasets provided by [19] . We set a sampling stride of frames to 8, 1, and 1 for 50 Salads, Breakfast, and GTEA, respectively. For the diffusion process, we set the entire time step S as 1000 [29, 54] , with a skipped time step for inference set to 25 [54] . For the anticipation mask M A , we set the observation ratio α ∈ {0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8}. For the random mask M R , we fix the size of patch w to 10 for both tasks and randomly select the number of masked patches N R to 25, 10, and 20 for 50 Salads, Breakfast, and GTEA. For the encoder and decoder, we adopt a modified version of ASFormer [65] with using relative positional bias where the maximum number of neighbor is set to 100. During inference, we set α to 1 and β to 0 for TAS [65, 19, 43] . For LTA, we set α ∈ {0.2, 0.3} and β ∈ {0.1, 0.2, 0.3, 0.5}, following the evaluation protocols [2, 20, 52, 25] . See Sec. F for more details.Table 1 and 2 present comparisons of the proposed method with state-of-the-art models on TAS and LTA across benchmark datasets, respectively. We report the performance of a single unified model on both tasks for each dataset. In Table 1 , ActFusion achieves state-of-the-art performance across benchmark datasets, where the average scores of the performance outperform all the other task-specific models. Compared to DiffAct, which mainly focuses on learning TAS, ActFusion demonstrates better performance, showing the effectiveness of joint learning TAS with LTA.EQUATIONTable 2 shows the result for LTA, with the table divided into two sections based on the dataset, and each section further divided into two subsections based on the input type. The first subsection shows the results using prediction results of the visual features [2] from [49] , while the second subsection compares the model using pre-trained I3D features [9] as input. Overall, ActFusion achieves stateof-the-art performance on two benchmark datasets, demonstrating the efficacy of joint learning for TAS and LTA based on the diffusion process. Note that we do not include the performance of ANTICIPATR [44] due to differences in evaluation ranges, as reported in [67] , where they employ different prediction ranges for evaluation. Tables 1 and 2 show that our single unified model outperforms task-specific approaches on both TAS and LTA tasks.We conduct comprehensive analyses to validate the effectiveness of the proposed method. In the following experiments, we evaluate our approach on the 50 Salads dataset. All experimental settings are the same as explained in Sec. 5 unless otherwise specified.Segmentation helps anticipation. To validate the impact of learning TAS on LTA, we conduct loss ablation experiments by removing the action segmentation loss on the observed αT frames. Table 3 shows the results. Specifically, we exclude the encoder loss L enc in Eq. 11, while omitting the decoder loss L dec in Eq. 12 on the observed frames. For simplicity, we denote the decoder loss for the observed frames and unobserved frames as L dec o and L dec u , respectively. Note that L dec u are utilized for effective anticipation learning. By comparing (1) and (3) in Table 3 , we observe a significant performance drop when removing the loss on the observed frames. We observe that applying segmentation loss on the observed frames in the decoder improves performance when comparing (1) and ( 3). The overall results show that learning action segmentation plays a crucial role in improving action anticipation.Anticipation helps segmentation. To validate the effect of jointly learning anticipation along with segmentation, we conduct ablation studies on anticipative masking M A in Table 4 . Comparing (1) and (2) in Table 4a , we find that using anticipation masking helps improve the performance of TAS. In comparison to the results in Table 4 , where segmentation greatly enhances anticipation, the performance improvement without using anticipation masking is slightly lower. Nevertheless, we find that anticipation does contribute positively to the overall metrics of segmentation.Ablation studies on masking types. To evaluate the effects of different types of masking on TAS and LTA, we conduct ablation studies in Table 4 . The performance on TAS and LTA are presented in Table 4a and 4b, respectively. We observe that anticipative masking M A plays a significant role in the joint learning of TAS and LTA, as evidenced by the substantial performance drop in LTA when comparing row (2) with the other rows in Table 4b . Random masking M R significantly reduces 4 . Furthermore, the use of boundary masking M B and relation masking M S is also essential for both tasks. Due to the limited space, we only report the performance on LTA when the observation ratio α is set to 0.3.Effects of learnable mask tokens. To verify the effects of learnable mask tokens m, we replace m with zeros that are not learnable. Table 5 shows the overall results. Comparing rows ( 1) and(2) in Table 5 , we find that using learnable mask tokens is more effective for both TAS and LTA. We conjecture that this is due to the mask tokens being embedded with the visual tokens within the encoder, which aids in effectively anticipating the invisible parts.Position of mask tokens. Instead of providing mask tokens as input to the encoder, we conduct experiments on providing mask tokens as input to the decoder, as shown in row (3) in Table 5 . In this experiment, only the visible frames are given as input to the encoder, and mask tokens m ∈ R 1×D are broadcasted to be concatenated to the embedded features E and reorganized to the original masked positions, similar to masked auto-encoder [28] . Comparing rows ( 2) and (3) in Table 5 , we find that using mask tokens in the encoder is more effective than using them in the decoder. We conjecture that embedding the mask token with the visual tokens via the encoder is beneficial for joint learning of TAS and LTA. Additionally, if mask tokens are provided as input to the decoder, they cannot receive the encoder loss L enc at all, which may degrade performance.The number of masked clips in random masking. To determine the number of masked clips N R in random masking, we conduct experiments by adjusting N R from 10 to 100 while fixing the window size Q of each masked clip N P to 10. Figure 3 illustrates the result, suggesting that employing an appropriate amount of masking is crucial.The number of inference steps in the diffusion process. We compare performance according to the number of inference steps of the diffusion process in Fig. 4 . We observe consistent performance increases as the number of inference steps increases. As the increasing step requires more computation, we choose to use 25 steps for inference in all of our experiments.Figure 5 presents qualitative results from ActFusion, evaluated on both TAS and LTA using a single model. The figure includes video frames, ground-truth action sequences, and predicted results for TAS and LTA. For LTA, only the visible parts (observed frames) are used as input. The results show that ActFusion effectively handles both visible and future segments, accurately classifying current actions and anticipating future ones. Additional results are provided in FiguresS3 and S4.Benchmark evaluations in previous work, including those of [20, 25] and our results presented above, have been typically conducted following the evaluation protocol of [2] where prediction length N A is set to βT in testing; β and T are the prediction ratio and the ground-truth video length, respectively. This can be seen as a leakage of ground-truth information in testing because the exact length of future actions is supposed to be inherently unknown during inference in real-world scenarios. In this subsection, we thus rectify the evaluation setting by determining the prediction length as rN O based solely on the number of observed frames N O , where r is a hyperparameter that adjusts the relative length of future predictions. We then train our method with this modified anticipation masking strategy, denoted as ActFusion † . In this experiment, r is set to 4.Using the rectified evaluation protocol, we compare ours with Cycle Cons. [20] and FUTR [25] , whose codes are available 1 . For a fair comparison, we modify both models to use the same prediction length rN O , denoted as Cycle Cons. † and FUTR † . The results are summarized in Table 6 where the numbers in parentheses indicate relative performance changes compared to those in Table 2 . Table 6 shows that all the methods exhibit overall performance degradation when ground-truth length information is not used. These observations reveal that previous results have benefited from the use of ground-truth length T , leading to information leakage in testing. Therefore, we suggest that future research avoid relying on this information for a more realistic and fair comparison. On the other hand, the results also show that our approach consistently outperforms the methods across benchmark datasets, demonstrating its effectiveness even without ground-truth prediction length information. Please refer to Sec. F for further details of the experimental setup.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes] Justification: We include the code and instructions for reproduction in the supplementary.The training and validation data is available online.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
conclusion: 
 We have presented ActFusion, a unified diffusion model that jointly addresses temporal action segmentation and long-term action anticipation in videos. The key to our method is anticipative masking, where learnable mask tokens replace unobserved frames, enabling simultaneous action segmentation of visible parts and anticipation of invisible parts. Our comprehensive experiments demonstrate that this unified approach not only outperforms existing task-specific models on both tasks but also reveals the mutual benefits of joint learning. Additionally, by evaluating our method both with and without ground-truth length information during LTA inference, we hope to motivate future research toward not using this information during testing. We believe that ActFusion demonstrates the potential of unifying complementary tasks in temporal action understanding, opening new directions for future research.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: We specify all the training details in the main and supplementary.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
acknowledgement: 
 This work was supported by Samsung Electronics (IO201208-07822-01) and IITP grants (RS-2022-II220959: Few-Shot Learning of Causal Inference in Vision and Language for Decision Making (50%), RS-2022-II220264: Comprehensive Video Understanding and Generation with Knowledgebased Deep Logic Neural Network (45%), RS-2019-II191906: AI Graduate School Program at POSTECH (5%)) funded by Ministry of Science and ICT, Korea. We thank reviewers for pointing out the issue of the evaluation protocol and providing insightful discussions.Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: We do not report error bars due to the limited computational resources.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes] Justification: We include the amount and type of resources used for training in Sec. F.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: We cite all the papers that are relevant to the code, models, and datasets. We include the license of each asset in Sec. F. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 13. New Assets Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [Yes] Justification: We include trained models in the supplementary for reproduction of our main results.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used.  
 
 Appendix: In this appendix, we offer detailed descriptions and additional results, which are omitted in the main paper due to the lack of space. We provide a detailed explanation of diffusion models in Sec. A, training and inference algorithms in Sec. B, more details of masking types in Sec. C, encoder and decoder layers in Sec. D, additional experimental results in Sec. E, datasets and implementation details in Sec. F, different evaluation setup of ANTICIPATR [44] in Sec. ??, codes in Sec. ??, more qualitative results in Sec. G, and discussions of limitations and broader impacts in Sec. H.In this section, we give an in-depth explanation of the diffusion models [29, 54] described in Sec. 3. The forward process adds noise to a data distribution x 0 ∼ q(x 0 ), following the noise distribution q based on the Markov property. Specifically, the forward process is defined by adding noise at an arbitrary time step s with a variance γ(s) as follows:EQUATIONEQUATIONwhere S represents the entire time step. We can directly obtain the noisy data distribution x s at time step s without iteratively applying q due to the Markov property:EQUATIONx s := δ(s)EQUATIONwhere δ(s) := 1 -γ(s), δ(s) := s i=1 δ(i), and ϵ ∼ N (0, I). Instead of using γ(s), 1 -δ(s) is utilized [29] . The posterior q(x s-1 |x s , x 0 ) conforms to a Gaussian distribution and is expressed in terms of mean ms (x s , x 0 ) and variance γ(s) using Bayes Theorem:EQUATIONwhere ms (x s , x 0 ) := δ(s -1)γ(s)EQUATIONEQUATIONWith a sufficiently large S and an appropriate variance schedule γ(s), the noisy data x S follows an isotropic Gaussian distribution. Consequently, if we know the reverse distribution q(x s-1 |x s ), we can sample x S ∼ N (0, I) and reverse the process to obtain a sample from q(x 0 ). However, since q(x s-1 |x s ) relies on the entire data distribution, we employ a neural network to approximate q(x s-1 |x s ) as follows:EQUATIONwhere m θ and Σ θ represent the predicted mean and co-variance, respectively, derived from the neural network. Predicting ϵ or x 0 in Eq. 16 is found to be effective [29] . In this work, we choose to predict x 0 using the neural network. i,a ̸ = 1), i ∈ {1, 2, ..., T } and a ∈ {1, 2, ..., K}, with T representing the number of frames and K denoting the number of action classes. Here, a specific action class a is randomly selected for the mask. The boundary mask M B hides visual features near the boundaries based on the soft ground truth B to manage the uncertainty associated with action transitions. Specifically, it is defined byM B i = 1( B < 0.5),where i ∈ {1, 2, ..., T }.During training, a masking type is randomly selected among five: no mask M N , anticipative mask M A , random mask M R , relation mask M S , and boundary mask M B , for each iteration as detailed in Sec. 4.4. During inference, the no mask M N and the anticipative mask M A are utilized following the evaluation protocol [19, 65, 43, 20, 52, 25] .We employ a modified version of ASFormer [65] utilized in DiffAct [43] as the baseline model. An encoder g consists of N E number of layers and a decoder h consists of N D number of layers as described in Sec.4.2. Figure S2 illustrates detailed operations of the encoder and the decoder layers. An encoder layer consists of dilated convolution followed by dilated attention, layer normalization, and a feed-forward network. Additionally, we add relative position bias B to attention scores to consider relative position relations among actions [48] . Similarly, a decoder layer comprises dilated convolution followed by dilated attention, layer normalization, and a feed-forward network. The output embedding E from the encoder is concatenated to the input of the decoder after dilated convolutions. Subsequently, they are embedded as queries and keys in the dilated attention operation. We refer the reader to [65, 43] for more details.We provide additional experimental results, maintaining the same experimental settings as described in Sec.5.2 unless otherwise specified. The evaluation is conducted on the 50 Salads dataset.Conditioning features. We investigate different types of conditioning features: masked features F ′ , output embeddings E from the encoder, and the encoder prediction A ′ . Table S1 shows overall results, where using E is more effective than using other features. The effectiveness of utilizing the encoder in our approach becomes apparent when comparing the first and second rows, where a significant performance drop is observed in the absence of the encoder. Since mask tokens replace visual tokens before going into the encoder, it is crucial for them to learn action relations through the encoder. Comparing the second and third rows, we also observe that utilizing features from intermediate layers yields slightly better results than using encoder predictions for both TAS and LTA.Position embedding. In Table S2 , we explore different types of position embeddings: relative position bias [48] , relative position embedding [53] , and absolute position embedding [15] . Comparing the first and second rows, we find that employing relative position bias enhances the overall performance in TAS and LTA. From the second and third rows, we observe that relative position bias is also more effective than using learnable relative position embedding. Using absolute position embedding leads to decreased performance in both TAS and LTA. We find that learnable embeddings often cause overfitting problems during training. As a result, we adopt relative position bias in our model.We conduct ablation studies on the loss functions: boundary loss, smoothing loss, and cross-entropy loss. Table R4 presents the results, demonstrating that the combination of bounding loss and smoothing loss is effective for both TAS and LTA. While the effectiveness of these losses in TAS is well-documented in previous research [18, 64, 42] , their impact on LTA has been less explored. Notably, the smoothing loss leads to significant performance gains in both tasks, indicating that smoothed predictions are beneficial.Effects of reconstruction loss L recon Masked auto-encoding is a technique used in training NLP models like BERT [15] and has recently been adapted to vision models [22, 28] . Inspired by this approach, we train our model to reconstruct input video features from the masked tokens as an auxiliary task. Specifically, we employ MLP layers on the encoder embeddings to reconstruct the input video features and apply mean squared error (MSE) loss L recon during training.Table S4 shows the overall results on both TAS and LTA tasks. In TAS, overall performance increases. We conjecture that reconstruction helps the model gain a deeper understanding of the underlying data structure and temporal dynamics by predicting the missing parts of the input. In LTA, we find that reconstruction is more effective on relatively short-term anticipation. Since short-term predictions are often based on more immediate context, there is less uncertainty. As a result, reconstructing masked features helps the model capture immediate patterns and transitions more accurately. However, for long-term predictions, as the model attempts to predict further into the future, the uncertainty increases significantly. Long-term predictions involve more variables and potential changes, making them inherently less predictable. This increased uncertainty might cause performance degradation, making reconstruction less effective for action anticipation.Datasets. The 50 Salads [58] dataset consists of 50 videos depicting 25 individuals preparing a salad. With over 4 hours of RGB-D video data, the annotations include 17 fine-grained action labels and 3 high-level activities. Notably, 50 Salads videos are longer than those in the Breakfast dataset, averaging 20 actions per video. The dataset is partitioned into 5 splits for cross-validations, and we report the average performance across all splits. The Breakfast dataset is under the license of Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License. The Breakfast [36] dataset consists of 1,712 videos featuring 52 individuals preparing breakfast in 18 different kitchens. Each video is assigned to one of the 10 activities associated with breakfast preparation, utilizing 48 fine-grained action labels to define these activities. The average video duration is approximately 2.3 minutes, encompassing around 6 actions per video. The dataset is divided into 4 splits for cross-validations, and we report the average performance across all splits. The Breakfast dataset is under the license of Creative Commons Attribution 4.0 International License. The GTEA dataset [21] Table S1 comprises 28 videos capturing 11 action classes related to cooking activities. Each video includes 20 actions on average, and an average video duration is about a minute. The dataset is divided into 4 splits for cross-validations, and we report average performance across all splits. The GTEA dataset is under the license of Creative Commons Attribution 4.0 International License.Cross-task generalization experiments. In Fig. 1-(c ), we conduct experiments on cross-task evaluations with DiffAct [43] , FUTR [25] , and TempAgg [52] . To evaluate DiffAct on LTA, we limit the input video frames to F 1:αT , with zero masks appended to future frame lengths concatenated to the encoder output embeddings, adhering to DiffAct's masking strategies. Subsequently, the decoder predicts future actions based on the observed video frames. To evaluate FUTR on TAS, we utilize the encoder of FUTR as an action segmentation model. Note that we report the performance of TempAgg on both TAS and LTA as provided in the original paper [52] .Evaluation without using ground-truth prediction length on LTA. Baseline models of Cycle Cons. [20] and FUTR [25] anticipate future actions and their corresponding durations. Then, the predicted durations are multiplied by the ground-truth prediction length βT to generate final predictions.In Table 6 , we adopt the modified prediction length rαT . Please note that we reproduce Cycle Cons. and utilize the official checkpoints of FUTR.Implementation details. We provide additional implementation details to complement those described in Sec. 5.2. Table S5 presents the specific hyperparameters used in our experiments for each dataset. All experiments are conducted on a single NVIDIA RTX-3080 GPU. We implement ActFusion using Pytorch [47] and some of the official code-base for DiffAct [43] 2 licensed under a MIT License.We provide additional qualitative results for both successful and failure cases in Fig. S3 and Fig. S4 , respectively. Figure S3 demonstrates the promising results on both TAS and LTA, showing the efficacy of joint learning these two tasks. Particularly, ActFusion successfully anticipates all the necessary actions to make a salad even in a longer sequence in Fig. S3d . In the failure cases, figure S4a highlights the importance of accurate segmentation on anticipation, where inaccurate action segmentation of the observed frames leads to wrong future anticipation. In Fig. S4b , ActFusion fails to detect the 'take knife' action class in action segmentation. However, the model anticipates missing action classes that are not observed in the observed frames. This implies that the model can infer missing actions based on the action relations of the observed and unobserved actions.In this paper, we have proposed a unified diffusion model for action segmentation and anticipation, dubbed ActFusion. We have demonstrated the effectiveness of the proposed method through comprehensive experimental results, showing the bi-directional benefits of joint learning the two tasks. In this section, we will discuss the limitations, future work, and broader impact of the proposed method. Limitations and future work. The proposed method shows the state-of-the-art results on TAS across three benchmark datasets. However, the frame-wise accuracy is slightly below compared to DiffAct [43] . We hypothesize that this discrepancy arises from the different masking strategies employed. In DiffAct, masking is applied to the output embeddings of the encoder, ensuring that the encoder always fully observes the visual features from the entire video. However, in our approach, masking is applied before the encoders, allowing for handling both visible and invisible tokens to unify the two tasks effectively. Consequently, the encoder may not always observe the entire visual features, potentially leading to slightly lower accuracy. This issue could potentially be addressed by employing reconstruction methods for masked features, similar to masked autoencoders [28, 22] . By further training the model to reconstruct the original features from the masked features, the encoder could be empowered to handle the masked features better. We leave this as our future work.In future work, additional activity information could be integrated, particularly focusing on segmentwise action relations [8] . Moreover, exploring weakly-supervised learning approaches [39, 49, 37] could be beneficial for further enhancing the capabilities of our method.Broader impact. To the best of our knowledge, we are the first to integrate action segmentation and anticipation within a unified model. We believe that our work lays the foundation for integrating these two tasks, offering substantial potential for real-world applications such as robots. , with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.The official code for Cycle Cons.[20] was obtained directly from the authors, and the official code repository for FUTR[25] is available at https://github.com/gongda0e/FUTRhttps://github.com/Finspire13/DiffAct 


Decision: All 3 reviewers were inclined to accept the paper. All reviewers agree the paper tackles an important problem in an interesting way and the results show that the method improves over previous SOTA significantly. The AC has read all reviews and, aligned with the reviewer recommendation, the AC is recommending the paper be accepted to NeurIPs.

Reviews: review: paper_summary: paper_summary: The author introduce a unified diffusion model for temporal action segmentation (TAS) and long-term action anticipation (LTA), dubbed ActFusion, where a single model is jointly trained to address these two problems effectively. A new anticipative masking is presented for the effective unification of two tasks, along with random masking to learn intra-action relations. ActFusion achieves the state-of-the-art performance on both TAS and LTA, demonstrating the effectiveness of joint learning of two tasks across standard benchmark datasets, 50 Salads and Breakfast, and GTEA.
strength_weakness: strength_weakness: * The methodology lacks novelty. The model architecture, loss function (Cross-Entropy Loss, Temporal Smoothness Loss, Boundary Alignment Loss) and mask strategy (no mask, relation mask, boundary mask) closely resemble those used in the Diffusion Action Segmentation (https://arxiv.org/pdf/2303.17959v2). The primary distinction in this work is only its extension to include the long-term action anticipation task, and the introduction of anticipative masking.
* The performance improvement relative to other state-of-the-art works is marginal. On TAS task, when compared to DiffAct, this model shows an approximate 1-point improvement across various metrics (F1, edit score, frame-wise accuracy) on different benchmarks (50 Salads, Breakfast, GTEA). 
* The model consistently achieves sub-optimal results when assessed using frame-wise accuracy as the metric on TAS task. As the authors point out, this could potentially be addressed by employing reconstruction methods for masked features. I am eager to see how these adjustments could enhance the model's performance.
* More ablation study for LTA task could be included, e.g., How important is past context for the models (\alpha) ? How far into the future can models predict (\beta)? Table 2 reports only a limited range of settings. A more thorough analysis on these aspects would be highly valuable.
questions: questions: * The approach is currently limited to predicting action labels within a closed set. To extend this work to predict open-set action labels, what can be done?
* While "segmentation helps anticipation" is evident from table 3, "anticipation helps segmentation" is considerably less significant in table 4. What's the reason behind this?
* If further training the model to reconstruct the original features from the masked features, would it also improve the LTA task?
* Paper writing
    * What does circle and triangle means in figure 1(c)?
    * In table2, why there are multiple underlined values (suppose to be second-highest value) in each column?
limitations: limitations: * Limitation
    * The author notes the sub-optimal performance when using frame-wise accuracy to evaluate the TAS task and suggests a potential solution, though its effectiveness remains unproven. I am eager to see how these proposed adjustments might improve the model's performance.
* Potential negative societal impact
    * There is no potential negative societal impact of this work.
paper_summary: paper_summary: This paper proposes a new unified diffusion model called ActFusion, which solves the tasks of Time Action Segmentation (TAS) and Long Term Action Prediction (LTA) in a joint learning framework. To unify the two tasks, the model effectively handles the visible and invisible parts of the sequence during the training phase; The visible part is used for observed video frames, while the invisible part is used for future expectations. The experiment showed that the model achieved state-of-the-art performance in standard benchmark tests of 50 Salad, Breakfast, and GTEA.
strength_weakness: strength_weakness: 1. What is the difference between the Diffusion model used in the paper and the DiffAct model? They appear to be similar overall, despite using different Masks to unify the tasks of TAS and LTA.
2. The paper employs so many loss functions for model training; the authors should analyze the impact of different losses rather than simply exploring the effects of the encoder and decoder.
3. Since the denoising process of the diffusion model is a time-consuming task, I am interested in the computational efficiency of the model proposed in the paper. The authors are advised to provide information on the model's GFLOPs, parameter scale, and inference time.
4. The authors need to further improve the interpretation of the figures, such as what the circles and triangles in Figure 1(c) represent.
questions: questions: 1. What is the difference between the Diffusion model used in the paper and the DiffAct model? They appear to be similar overall, despite using different Masks to unify the tasks of TAS and LTA.
2. The paper employs so many loss functions for model training; the authors should analyze the impact of different losses rather than simply exploring the effects of the encoder and decoder.
3. Since the denoising process of the diffusion model is a time-consuming task, I am interested in the computational efficiency of the model proposed in the paper. The authors are advised to provide information on the model's GFLOPs, parameter scale, and inference time.
4. The authors need to further improve the interpretation of the figures, such as what the circles and triangles in Figure 1(c) represent.
limitations: limitations: In summary, the authors have creatively used different Masks to unify the tasks of TAS and LTA with a single model and achieved significant results. However, compared to existing models, there does not seem to be much modification, highlighting a disadvantage in innovation.
paper_summary: paper_summary: This paper extends DiffAct to perform both action segmentation and action anticipation. An anticipative masking with a learnable mask token is proposed. Experiments are conducted on the three common benchmark datasets.
strength_weakness: strength_weakness: 1. The major technical problem is that the proposed method assumes the ground truth video length is known for action anticipation at the test time. Without the ground truth video length T, the anticipative mask M^A can not be constructed during the inference time for anticipation. This is also shown in the code provided, where 'full_len' is input into the ddim inference function. This is problematic, conflicting with the goal of anticipation, and leading to incomparable experimental results.

2. The technical novelty is limited. The main contribution is extending the DiffAct method with an anticipative mask, while other modules are from existing methods. But given the contribution of unifying the segmentation and anticipation tasks, this is not a deciding factor for me.

3. It will be better to conduct experiments on Assembly101. Given the small data size and the saturated performance on GTEA, I would recommend a transition from GTEA to Assembly101 for this task.

Minor:
- In Figure 1c, what do the triangle and the circle mean?
- In Table 1, some recent methods are missing, such as MVGA ICCV23 and RTK ICCV23.
questions: questions: NA
limitations: limitations: The limitations were briefly discussed.

Comments: title: A gentle reminder
Test Data Leakage and Difficult Case
Response
comment: Thank you for the response. We are glad to hear that most of the concerns have been addressed by our rebuttal. 
We would like to thank reviewer FW3D once again for the insightful comments for extensive experiments and directions for our work. We make sure to include all experimental results in the final manuscript.
Thank you to the authors for providing detailed explanations and conducting additional experiments. These have addressed all of my questions and concerns. Please ensure to include these in the final version of the manuscript. I will be adjusting my rating accordingly.
Dear reviewer FW3D, 

We'd like to thank again for your effort and time dedicated to our submission. We've addressed your concerns in our rebuttal, and it would be very helpful if you could give us any further thoughts and update your scores before the author-reviewer discussion phase ends. Your opinion would be invaluable to us in improving our work, and we would be glad to respond further to your questions. Thank you for your consideration.

Best regards,  
Authors
We thank reviewer uA7n for the motivating feedback. We are pleased to hear that most of the concerns have been addressed by our rebuttal. The results discussed in the rebuttal will be included in the final manuscript.
We sincerely appreciate the reviewer’s constructive feedback, which has been invaluable in raising an important issue within the community and guiding us to improve our submission. We will ensure that all experiments in our final manuscript are fully updated without the use of ground-truth length during inference.

As the reviewer zKDZ mentioned, we also believe that revealing this critical issue and rectifying it would be a significant contribution to the community. We will thoroughly analyze and bring to light the issues shared by previous methods (at least, including [19, 24]), ensuring that all models are compared under adequate and fair conditions.
Thank you for your detailed rebuttal. The author has clarified most of my concerns. I keep my score as weak accept.
Thanks very much. I really appreciate the additional experiments. It is helpful to see the new results without using ground-truth length, which are quite reasonable.

It is surprising that all previous methods have utilised the ground-truth length during testing. I would flag this as a critical issue worth the community's attention. Revealing this issue using the above experiments might be a significant contribution to the community.

Therefore, I have greatly increased my score, on the condition that the additional experiments will be included in the later version. And all the other experiments should also be 'totally' updated to this no ground-truth version. However, this might be too substantial and there is no mechanism to guarantee that. 

Overall, this is really a difficult case. Therefore, I have also changed my rating to the lowest confidence, and I would leave this to the chairs for the final judgement.
### **[W1: Using the ground-truth video length during inference]**

Thanks to your response, now we fully understand the point regarding the use of the ground truth video length during inference. To address this concern, we conducted additional experiments where no ground truth length is used in testing; following your suggestion, in testing, we fixed the length of future frames (i.e., mask tokens) to the maximum number of future frames in the training set.

Experimental results in this setting on the 50 Salads dataset are reported in Table R6. In this table, the column ‘use of GT length’ indicates if each model exploits ground-truth length during testing. We found that, following the first paper introducing long-term dense action anticipation [2], all previous methods utilize the ground-truth length during testing, except for those whose codebases and/or inference setups are not available [25, 51, 65]; we marked these methods 'unknown' in the column.

In the table, **ActFusion*** represents **our original model using a fixed number of mask tokens without retraining.** To ensure a fair comparison, we applied the same testing scheme to Farha et al [19] (denoted by Farha et al*) and FUTR [24] (denoted by FUTR*), *both of which originally utilize the ground-truth length during testing*. In this setting, ActFusion* still outperforms Farha et al* and FUTR*, and as the reviewer expected, all of these methods perform worse.

This performance drop is due to the use of the ground-truth video length in training, which results in a discrepancy between the training and inference setups. To mitigate this issue, **we retrained our model while fixing the number of mask tokens to cover the maximum number of future frames in the training set.** The retrained results, presented in the last row of Table R6 and denoted as **ActFusion†**, achieve the state of the art in long-term action anticipation (LTA). We observed that fixing the number of mask tokens brings performance gain when the prediction ranges are relatively short, as the model benefits from more stable predictions. However, for longer predictions, particularly when the prediction ratio is set to 0.5, we observed performance degradation compared to ActFusion. This degradation is probably due to the fact that it becomes more difficult for the model to determine the end of an activity. Nonetheless, these results demonstrate that our method can be flexibly adapted to different numbers of mask tokens, ultimately achieving the state of the art in LTA.

We sincerely hope this clarification addresses your concerns. We believe that the experimental setup you suggested will contribute significantly to the field by providing more realistic and reasonable evaluation protocols, and we will include all of the above results in the revision.

**[Table R6. LTA results with and without using ground truth length]**

| method            | use of GT length | $\alpha=0.2,\beta=0.1$ | $\alpha=0.2,\beta=0.2$ | $\alpha=0.2,\beta=0.3$ | $\alpha=0.2,\beta=0.5$ | $\alpha=0.3,\beta=0.1$ | $\alpha=0.3,\beta=0.2$ | $\alpha=0.3,\beta=0.3$ | $\alpha=0.3,\beta=0.5$ | Avg.  |
|:-------------------|:-------------:|:------------------------:|:------------------------:|:------------------------:|:------------------------:|:------------------------:|:------------------------:|:------------------------:|:------------------------:|:---------:|
| Temporal Agg. [51]| unknown| 25.50| 19.90| 18.20| 15.10| 30.60| 22.50| 19.10| 11.20| 20.26|
| A-ACT [25]| unknown| 35.40| 29.60| 22.50| 16.10| 35.70| 25.30| 20.10| 16.30| 25.13|
| Object Prompt [65]| unknown| 37.40| 28.90| 24.20| **18.10**| 28.00| 24.00| **24.30**| 19.30| 25.53|
| Farha et al. [19] | ✓| 34.76| 28.41| 21.82| 15.25| 34.39| 23.70| 18.95| 15.89| 24.15|
| Farha et al.* [19]| -| 29.07| 23.83| 20.49| 12.77| 26.51| 17.78| 14.35| 11.19| 19.50|
| FUTR [24]| ✓| 39.55| 27.54| 23.31| 17.77| 35.15| 24.86| 24.22| 15.26| 25.96|
| FUTR* [24]| -| 28.84| 20.01| 16.65| 11.37| 22.48| 16.49| 13.21| 9.21| 17.28|
| ActFusion (ours)  | ✓| 39.55| 28.60| 23.61| 19.90| 42.80| 27.11| 23.48| 22.07| 28.39|
| ActFusion* (ours) | -| 34.50| 26.17                   | 20.27                   | 11.87                   | 34.58                   | 22.75                   | 17.31                   | 11.33                   | 22.75|
| ActFusion† (ours)| -           | **41.30**               | **30.83**               | **24.40**               | 16.10                   | **41.70**               | **28.08**               | 22.48               | **19.56**               | **28.06**|

- In the table, the bolded values represent the highest accuracy among the models that do not use ground truth length, ensuring a fair comparison.
Thanks very much for the response.

W1: I am not convinced. 

This is not an evaluation choice as argued by the authors. This is a test data leakage issue. For any machine learning model, you should not use any ground truth when obtaining predictions on test data. 

One possible solution might be using the mean length of training videos as the 'full_len' during testing. But I guess this will lower the results. 

As for previous methods mentioned by the authors, I did not check their code. Did other previous methods except for those mentioned also use this ground truth? Even if ground truth was used in some of previous codes, I would consider this as an issue to be fixed in the following works rather than a convention to be inherited. 

W2: I appreciate the novelty of the unification, but not the methodology. This is subjective though.

W3 and minor have been addressed.