OnlineTAS: An Online Baseline for Temporal Action Segmentation
Authors: Qing Zhong
Guodong Ding
Angela Yao
Venue: NeurIPS 2024 poster (2024)
Keywords: Temporal Action Segmentation
Video Understanding
Primary area: machine_vision

Abstract: Temporal context plays a significant role in temporal action segmentation. In an offline setting, the context is typically captured by the segmentation network after observing the entire sequence. However, capturing and using such context information in an online setting remains an under-explored problem. This work presents the first online framework for temporal action segmentation. At the core of the framework is an adaptive memory designed to accommodate dynamic changes in context over time, alongside a feature augmentation module that enhances the frames with the memory. In addition, we propose a post-processing approach to mitigate the severe over-segmentation in the online setting. On three common segmentation benchmarks, our approach achieves state-of-the-art performance.

OnlineTAS: An Online Baseline for Temporal Action Segmentation
Qing Zhongl.24
Guodong Ding
Angela Yao²
Universsity of Adelaide qing. Lang.zhoor zhongoddelaied edu. au
National University of Singapore
[dinggd, -comp.. nus edu
Abstract
Temporal context plays a significant role temporal action segmentation. offline setting, the context is typically captured by the segmentation network after observing the entire sequence. However, capturing and using such context information in online setting remains under-explored problem. This work presents the an online framework for temporal action segmentation. At the core the framework is adaptive memory designed to accommodate dynamic changes in context over time, alongside a feature augmentation module that enhances the frames with the memory. In addition, we propose post-processing approach mitigate the severe over-segmentation in the online setting. On three common segmentation benchmarks, our approach achieves state-of-the-ar performance.
Introduction
This work addresses online temporal action segmentation (TAS) untrimmed videos. Such videos typically feature procedural activities consisting of multiple actions steps in loose temporal sequence to achieve goal ( For example, "making coffee" has actions: 'take cup' 'pour coffee', 'pour water', 'pour milk', 'pour sugar' and 'stir coffee'. Standard TAS models [1]
25 37
offline and segment-only videos of complete procedural activities. An online TAS model, in contrast, segments only up the current time point and does not have access to the entire video and, therefore, the entire activity.
Online TAS faces challenges similar to other online tasks 44 46 in establishing scalable network that can retain useful information from an ever-increasing volume of data and facilitate effective retrieval when required. Additionally, over-segmentation is common issue for offline TAS, where the segmentation model divides an action into many discontinuous sub-segments, leading to fragmented outputs. This issue is exacerbated in the online setting, partial data at the onset of an action may lead to erratic predictions and increased over-segmentation.
Most relevant to our task is online temporal action detection (TAD) 43 Online TAD aims to identify whether an action is taking place and the action category. TAD targets datasets like THUMOS [6), TVSeries (5), and HACS Segment [48]. Among these, 90.8% videos of THUMOS [66 feature only multiple instances of the same action, while TVSeries [5] comprises diverse yet independent actions (e.g., 'open door', 'wave" and 'write") in one video. These actions do not necessarily correlate with one another or impose specific temporal constraints. As such, a direct adaptation of popular online TAD approaches like LSTR [4 and MAT 41] to online segmentation is non-ideal. For instance, these models encode temporal context with a fixed set of tokens, which may limit their capability to handle the relations of procedural videos. Furthermore, these models are typically trained to prioritize frame-level accuracy while neglecting temporal continuity, which invariably leads over-segmentation.
Equaal Contribution. The work done when Qing Zhong intern in National University of Singapore.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

To address the online action segmentation task, this work proposes a novel framework centered on a context-aware feature augmentation module and an adaptive memory bank The memory bank, per-video, tracks short-term and long-term context information. The augmentation module uses an attention mechanism allow frame features to interact with context information from the memory bank and integrate temporal information into standard frame representations. Finally, we introduce a post-processing technique for online boundary adjustment that imposes duration and prediction confidence constraints to mitigate over-segmentation.
Summarizing our contributions, 1) We establish an online framework for TAS; 2) We propose feature augmentation module that generates context-awand representations by incorporating an adaptive memory, which accumulates temporal context collectively. The module operates on frame features independently of model architecture, enabling flexible integration: 3) We present a simple post-processing technique for online prediction adjustment, which can effectively mitigate the over- segmentation problem; and 4) Our framework achieves the state-of-the-ar online segmentation performance on three TAS benchmarks.
2
Related Work
Online Action Understanding. Many video understanding tasks, such as action detection
44
42
2 and video instance segmentation [15 46 47), have been explored in online contexts. For online action detection, videos are classified frame by frame without access to future frames. Specifically, LSTR [44] employs novel memory mechanism to model long- and short-term temporal dependencies by encoding them as query tokens. Follow-up works feature a segment- based long-term memory [1 and fusing short- and long-term histories via attention [2].
compression
However, the videos in the datasets commonly used in online TAD contain independent actions [5] or sequences of limited actions [16), thus lacking temporal relations between the actions. In contrast, TAS deals with untrimmed procedural videos, where such relations are more prominent and may span over long temporal durations. There is also growing trend in online TAD models to use action anticipation as an auxiliary task to enhance action modeling [2] [4]. In our online segmentation task, we do not assume the availability of such information.
Temporal Action Segmentation. In TAS 6 methods vary by their level of supervision, including fully [IT 20
27 semi-supervised 8 36), weakly [9] 13 22 30 29 31 21], and unsupervised [19 where procedural activities are learned sequentially. However, all existing works are offline, and complete video sequences can be used for inference. In contrast, our approach functions within an online setup. The most related work [33 investigates online TAS in a multi-view setup and leverages the offline model to assist online model learning. Furthermore, it uses the Trame-wise multi-view correspondence to generate pseudo-labels for action segments. In contrast, we do not assume the availability of multi-view videos nor require assistance from a pre-trained offline model.
34
32
10 setups. An emerging direction to learn TAS incrementally
Post-processing for Action Segmentation. Post-processing methods are either rule-based or leverage graphical modelling. Rule-based approaches [28] 35] 40 38) apply predefined rules smooth out short-duration predictions that are unlikely given the context. Graphical modelling approaches use Conditional Random Fields (CRFs) [7 30
model the relationships and transitions between consecutive actions. Online methods need post-processing to alleviate over-segmentation and help locate the action boundaries.
3 Online Action Segmentation
have demonstrated that the scope of the temporal context significantly influences the performance of (offline) TAS models. This motivates our two lines of inquiry for our online setting: 1) how to consolidate temporal context over an extended period, and 2) how to enrich the frame representations with context to benefit the segmentation. This work introduces a context-aware feature augmentation module (Sec. 3.2) alongside an adaptive context memory (Sec.) 3.3 to tackle the above questions.
Previous studies 1

---

3.1 Preliminaries
Consider an untrimmed video v == f2 of 'T frames, where 2; E R the per-extracted frame feature at time t and D is the feature dimension. An action segmentation model partitions into N contiguous and non-overlapping temporal segments {s; (yr, 4.) ]= corresponding to actions E y/ present in the video, where l indicates the segment length and y defines the action space [6] widely adopted strategy for TAS to design segmentation model .M that predicts the action label y/ for each frame Tt, akin to frame-wise classification. In the offline setting [11, 45] 25], the per-frame prediction Dt is based on the entire video sequence. The online setting uses only frames up to the current prediction time t, without access to future frames. Comparatively:
Dettine
arg max Pr (yh (1:T) and gonline arg max pe (3h (133).))) NEY
(I)
ycy
where Pt E
R V1 is the estimated action probability for frame 2t. Most existing offline TAS
works classification which is balanced by a smoothing loss (Lum) that encourages smooth transitions between consecutive frames:
23 train the segmentation model M using a cross-entropy loss (Las) for frame-wise
1 T
C
log(pc(u:))
Tp
.y
Cub
{ A A St
where Σ
and Ary pl logpt-1(y))
:otherwise
In this paper, we opt for the widely recognized convolution- based architecture MS-TCN | foundational framework. This choice is driven by its relatively lower computational requirements than the attention- diffusion-bart mode to an online mode of the segmentation model M is to substitute standard convolutions with causal convolutions. Causal and standard convolutions differ in their receptive field in that causal convolutions consider only past and present inputs while standard convolutions may incorporate both past and future inputs within kernel. Mathematical details and illustrations of the two are shown in the Appendix.
models 45 25]. A straightforward transition from offline
3.2 Context-aware Feature Augmentation
The context-aware feature augmentation (CFA) module generates enhanced clip-wise features through interactions with temporal context captured by an adaptive memory bank. The module operates on clip-wise basis. During training. video is split into K non-overlapping clips {cs f== Each clip has a window size w and is sampled from v with a stride of á = w, where the final clip сg is padded if |cx| < w. The CFA module integrates the original pre-extracted frame features Ck (mm)) (x= with temporal context to produce a context-enhancee version of representations
C = {ir) Like [44 41), our CFA module is also equipped with a simultancously updated memory bank M2 as a context resource for feature augmentation. The memory bank is further described in Sec. 3.3
At each step kt context is accumulated by feeding eg through a lightweight GRU The GRU is reliable in capturing information over long video sequences (24). The clip is then passed through a context aggregation block to be augmented. The context aggregation block incorporates the GRU features & CCU with the memory state Ma-1 from the previous step for / iterations. Concretely, we pass c through a self-attention (SA) block to encourage information exchange with the local clip window. Additionally, we leverage a Transformer decoder (39 to achieve a more effective memory encoding M i.e.,
to obtain GRU
cK SelfAtt(CCRR)) and MP TranssDecoder(MM---. GRU GRU Ck
The outputs from the self-attentio module (cSA) and the transformer decoder (c are then combined with cross attention (CA) and merged with ck produce the context-augmentted features:
č = CrossAttt(c M + GRU
(4)
The detailed formulas of SelfAutn(. Decoder),,,,( CrossAutn( are given the Appendix. An illustration of our CFA module is provided in Fig. [

---

Ck-1
2w
CFA
CFA
CFA
Adaptive
Adaptive Memory Bank
CA
Adaptive
Memory Bank
KA vA
Memory
Bank
Trans.
Trans. Decoder
Mx-2
Decoder
SA
MR-1
Mr
K v4
GRU
GRU
GRU
x
Ck-1
Ck
Figure 1: Context-aware Feature Augmentation (CFA) module. CFA takes as input a video clip ck of length w, augments it with temporal information captured in adaptive memory bank M3, and outputs an enhanced clip feature č₁. is the number of iterations of SA, TransDecoder, and CA.
3.3 Adaptive Memory Bank
In a similar spirit with 44, our memory is designed to account for both short- and long-term context, iee,, M [M Mfshor Short-term memory helps capture the local action dynamics while long-term memory retains information across extended durations important for TAS 13
Short Memory M Given that our enhancement module works on per-clip basis with temporal stride w, we directly regard the enhanced feature Čk-1 from the last clip as the short-term memory, ie., Mahon Ck-1 IR Dw
Long Memory Mlong We update our long-term memory with information from processed clips. Specifically, apply convolutional layer top of our context-enhanced representation Ck2 where G% R PM to collapse the temporal dimension and yield a memory token m, Conv1D(ãt) E R... This memory token is then appended to the current long-term memory.
Algorithm Adaptive Memory Update
Require: {c+] W
Initialize M 2: for k E [1...] {] do
C1, Monng Ø
CFA(CH, Mr-1)
4: my = CD (x))
if lelen (M
w then
Adaptive Memory Update. The memory is updated whenever a new clip is processed. In practice, we constrain the total footprint of both short- and long-
M Alang concat
M long
k- 17 m.k.
else
8:
Jony concat [M long
:], mp)
term memory to match the size the processing clip,
i.e., M I Dw At the beginning of each sequence, the memory bank is initialized with short-term infor-
end if
10:
M Ō [len M long
mation only, ie., M = Mchort = Ci and M = Ø. As more clips are processed, we gradually increase the budget to accommodate longer-term information.
11:
Mr [M Msshom
12: end for
However, in anticipation of long draining the entire budget in prolonged sequences, we cap its utilization at a maximum of two-thirds of the total budget. In instances where this threshold is exceeded, the earliest token is discarded, i.e.:
concat( M long
M my)
len ( < Zw
M long
ccocaa((M [1:],mx] m4)
otherwise
The remaining budget is allocated for the short-term information accordingly:
M é [len (
(6)
As video progresses, our feature augmentation module (Sec. 3.2 receives more longer-term context
while emphasizing only shorter and more relevant short-term information. This adaptive approach

---

Output y:
Step :
Je
Step k++1:
J6++1
(a) Online Inference (6=1)
(b) Semi-online Inference (6 m)
Figure Two inference types. a) Online inference samples clips with stride and only preserves the last frame prediction, while b) Semi-online inference samples non-ov non-overlapping clips with stride ò and all predictions are preserved.
enables the context memory to flexibly shift its attention between short and long-term information as the video progresses. Algorithm Algorithm[ssummaree summarizes the update mechanism.
Discussion. Our module integrates context memory top of GRU layer. While the GRU captures temporal dependencies, it may struggle, especially in thousands- long sequences common in TAS. The context memory supplements the GRU by allowing selective access and updates, thereby enabling the retrieval and manipulation of long-term information. Such a design is supported by our empirical study that the explicit memory can extend the capacity of the GRU's internal state. Supporting ablations are found in Sec. 4.1
3.4 Training and Inference
Training. Our final online segmentation model constructed by combining
Algorithm Post-processing for Online TAS
CFA module with a single-stage TCN []] with causal convolutions. Specifically, TCN takes as input the enhanced repre- sentations ē and maps them to the same labels for сg- We train the framework end-to-end with the loss function formulated in Eq. (2), but on a clip basis with T replaced by
Compute min T Tmax
2: Initialize { 0
for each frame do
4: if q: < and < Emia then
97 97-1
6:
l = f+1
else
celip Σ log(p)())) + A
Σ 2 1.3
8:
0f Ôt
10: end if
we set = 0.15 following I Like
training
11:
end for
on clip basis provides better efficiency.
Inference. We present two distinct inference ap-
proaches by manipulating the clip sampling stride parameter 5. The first mode of inference, referred as online, characterized by setting = 1. such a setting, a video clip of frames is processed, with emphasis placed solely on the prediction derived from the final frame and rest are discarded. This facilitates the scenario when frame-by-frame prediction is preferred. The alternate mode of inference, termed semi-online, adheres the training regime by setting 8 = In this mode, video clips are processed, and the dense predictions generated across all w frames are preserved as final output. An illustration of two modes of inference is provided in Fig.
3.5 Post-processing
Our intuition is that a valid action segment should not fall below a minimum length threshold unless there is high confidence in the prediction to justify change the action class. Specifically, we consider the maximum softmax probability of a prediction as its confidence measure, denoted as d: = max(p:), ). for frame V,, as q: to some extent indicates its reliability. prediction considered "unreliable" if its confidence measure scores below certain threshold 0, i.e., < 0. For the frame with "unreliable" prediction, we disregard its current prediction and assign the action label of its proceeding frame 97-1- when the previous action segment is shorter than the minimum length.
5

---

GTEA
12
50Salads
38
Breakfast
18
p.p.
Acc
Edit
FI @
[10,
25, 50]
Ace
Edit
Fl @
(10, 25,50) 50)
Acc
Edit
F1 @
[10, 25, 50
Online
74.4
66.6
73.9
70.3
57.2
75.2
19.6
26.8
24.4
19.6
55.3
18.7
15.1
11.7
8.3
72.1
71.9
79.2
77.4
64.1
75.1
68.5
74.1
70.6
60.4
52.3
54.7
52.0
43.2
29.8
Ours
76.2
63.5
72.6
68.3
58.8
80.9
28.8
36.1
31.0
23.3
56.7
19.3
16.8
13.9
9.3
v
74.4
70.3
78.5
76.4
67.7
77.7
71.5
77.7
74.6
64.1
52.9
55.7
54.8
45.8
30.5
Semi-online
TCN
75.8
66.8
74.3
71.5
60.3
29.0
38.5
35.5
28.3
55.7
18.6
15.4
9.0
73.5
75.4
80.3
76.9
66.6
76.7
69.2
73.1
70.5
62.8
52.5
54.0
53.1
44.5
29.6
Ours
77.1
68.1
76.7
73.5
63.9
82.4
32.8
43.0
41.1
34.7
57.4
19.6
17.8
14.8
10.1
76.0
79.7
84.9
81.4
69.2
79.4
75.0
82.5
80.2
68.0
53.8
57.5
56.4
47.3
31.4
Table I:
Performance our approach on three TAS benchmarks under two inference mode, i.e.,
online and semi-online. Post-processing is indicated by p.p..
Otherwise, we retain the original prediction. We set the length threshold fmin = x Tima with (0, 1), in proportion to the longest video duration Tmus training set.
Our post- sest-prcessing mitigates the over-segmentation by adjusting action boundaries according to
network predictions and action length, which is very efficient compared to B22 10 that calculates frame similarities. The procedure for post-prost post-processing is illustrated in Algorithm
Experiments
Datasets: We evaluate our model three common TAS datasets. Breakfast [8 comprises in total 1,712 videos performing ten different activities with 48 actions. On average, each video contains six action instances. 50Salads [38 has 50 videos with 17 action classes. GTEA [2] contains 28 videos of seven kitchen activities composing 11 different actions. We use common 13D features [ as input.
Evaluation Metrics: Standard evaluation metrics for TAS are reported for online setting, which includes frame-wise accuracy (Acc), segmental edit score (Edit), and segmental F1 scores with varying overlap thresholds 10%, 25 %, and 50%.
Implementation Details. In CFA, we stack Transformer decoder layer with heads, 2 Swin [26 self- and cross attention with 4 heads. We use single-stage TCN segmentation backbone and sample non-overlapping clips i.e., = w for efficiency. We train the model end-to-end with learning rate of 5e of total 50 epochs. Detailed hyperparameter settings can be found in Appendix.
4.1 Experiment Results
Effectiveness. We report the overall performance for online and semi-online inference (see Sec. 3.4 in in Table I Our baseline is a single-stage causal TCN and we build our framework on top of it. Across all three datasets, the integration of our CFA module leads to a consistent boost in the segmentation performance. Specifically, our approach gained 5.7% (75.2% vs. 80.9%) in Acc and 9.2% (19.6% vs. 28.8%) Edit on 50Salads. While the improvements on other datasets are not as significant, they still show effectiveness, with a margin of about 2%. Generally, semi-online inference achieves better performance over online across metrics. Such improvement likely because clip-wise prediction better preserves the local temporal continuity of labels compared to step-by-step single frame prediction.
Comparing across the metrics, segmental scores appear to be significantly low. On breakfast 18 with our online inference, frame- wise accuracy of 56.7% only corresponds 9.3% Fl score with 50% IoU. Such score indicates severe semeenemmntttion issue and necessitates an effective post-processing. However, significant performance increase observed on Edit and scores after our proposed pose-processing. For example, the same Fl score increases to 30.5% %, tripling its original value. Although post-processing could lead to a slight decrease in accuracy, it demonstrates great effectiveness in mitigating the segr-segmnntttion problem.

---

GRU
CFA
Mem.
Acc
Edit
F1
@
[10,
25, 50
I
Acc
Edit
F1
@
[10,
25, 50]
75.2
19.6
26.8
24.4
19.6
78.1
27.1
37.9
34.7
26.7
1
79.1
29.0
38.5
35.5
28.3
V
76.2
22.3
30.1
27.0
21.9
2
79.6
30.7
40.7
38.2
31.4
V
79.1
29.0
38.5
35.5
28.3
3
79.5
28.5
37.2
36.1
29.0
/
78.9
29.2
38.7
35.1
28.8
4
79.1
29.2
39.1
36.3
30.5
/
82.4
32.8
43.0
41.1
34.7
5
79.2
30.8
39.1
37.7
30.6
Table Ablation study of module components on 50Salads [38).
Table 3: Effect of interactions
Ablation study. 2 evaluaate the components in our CFA module. The first row is our single-layer causal TCN baseline with strong frame-wise accuracy poor segmental metrics. The GRU boosts segment metrics (7-11 () over the baseline, showing its ability to accumulate context information. While CFA using the current clip as pseudo memory predictably leads to performance drop (5%) compared GRU due to lack of any context information. Combining either GRU or adaptive memory with our CFA achieves very close performance (rows and 5), highlighting the importance of the context information for TAS. The complete model yields the best performance and boosts Acc by 7% and average segmental scores by 15.3%. This validates the complementary effect GRU's internal state and our explicit memory design.
Number of layers in CFA. Table Jexplores the interaction iterations 7 CFA. The results indicate that the performance is not significantly affected by the number of iterations. In practice, we set the number of iterations 2, as achieves good balance between performance and efficiency.
M
Mlong
Acc
Seg.
w / len(M)
16
32
64
128
256
80.3
36.7
Acc
79.8
81.4
81.6
82.4
80.7
80.4
36.4
Seg.
35.1
36.5
37.6
37.9
38.2
82.4
37.9
Table 5: Effect of clip size and memory length.
Table Effect of memory composition.
"Seg." " indicates the mean of Edit and F1 scores.
Memory composition lo))). We assess the impact of memory types and present results in Table It shows comparable performances for each memory type when considered individually. However, the combination of both yields 2% improvement Ace and 1.5% for averaged segmental metric, suggesting the significance of incorporating diverse memory for TAS.
Clip size / Memory size len(M), In our implementation, we set clip size and memory size to be equal and we report its influence on performance in Table 5 It shows larger clip size leads to better segmental results; this is because temporal continuity can better modeled with longer clips for learning. However, the information of short actions could be diluted when compressed to form the memory token if the window size is too large. For memory sizes of 16, 32, and 64, the earliest memory are discarded as the average length of 50Salads is 55.8k frames. With the size reducing, the performance gradually drops and reaches the lowest 79.8% Acc compared to the peak of 82.4%. Note that with the memory size set to 16, our approach only retains long-term information from up to 192 frames, 30x less than the average video length.
Post-process pest-procsssing hyperparameters. Two hyperparameters are defined in our post-processing: confi- dence threshold and the minimum segment length (ii- We vary its scaling factor assess l In Ta increasing 0 greatly enhances the segment results, with a 18. 1% increase observed when 0.7. Although the accuracy tends to decrease as 0 becomes larger, the drop not substantial (3.1%) compared to the improvements in segmental results. While Table 7 shows the segmental performance stops increasing and stays stable when a > Tē with fixed confidence score = 0.9. In conclusion, employing a higher confidence threshold can help better mitigate the over-segmentation because it makes more sense to prioritize preserving the continuity of a segment that includes frames with highly confident predictions given fixed length budget.
7

---

1
0.3
0.4
0.5
0.6
0.7
0.8
0.9
0
64
32
16
Acc
82.5
82.6
82.6
82.6
81.2
81.1
79.4
Acc
79.6
79.4
79.4
79.4
79.4
Seg. 70.9
74.7
76.4
76.4
76.4
Seg.
50.9
51.8
51.9
51.9
69.0
73.3
76.4
Table 7: Effect of minimum length factor 0
Table Effect of confidence threshold
(0)
with 0 0.9.
GTEA 12
50Salads
38
Method
Acc
Edit
F1 @
{10, 25, 50)
Acc
Edit
F1
@
[10, 25, 50
MS-TCN 11
78.7
84.0
88.3
86.6
72.8
81.2
65.8
72.8
70.4
61.7
offline
MS-TCN p.p.
78.7
85.2
89.6
88.3
73.3
80.4
74.1
82.0
79.2
70.2
ASFormer 45
79.7
84.6
90.1
88.8
79.2
85.6
79.6
85.1
83.4
76.0
DiffAct 25
82.2
89.6
92.5
91.5
84.7
87.4
88.9
90.1
89.2
83.7
LSTR [4]
63.7
33.2
41.5
37.7
25.0
60.5
5.0
8.2
6.6
4.1
Causal TCN
74.4
66.6
73.9
70.3
57.2
75.2
19.6
26.8
24.4
19.6
online
Ours online
75.8
66.8
74.3
71.5
60.3
79.1
29.0
38.5
35.5
28.3
Ourgonline
p.p.
73.5
75.4
80.3
76.9
66.6
76.7
69.2
73.1
70.5
62.8
Ourg semi
77.1
68.1
76.7
73.5
63.9
82.4
32.8
43.0
41.1
34.7
Ourgsemi p.p.
76.0
79.7
84.9
81.4
69.2
79.4
75.0
82.5
80.2
68.0
Table Comparison with the state-of-the-ar methods on GTEA and 50Salads.
4.2 Comparison with State-of-the-Art Methods
Tables 8 and 9 compare approach against the state-of-the-art TAS approaches all three benchmarks. Due to the absence of dedicated online TAS methods, we benchmark against the online TAD approach LSTR [44]. We train LSTR TAS datasets using the official code implementation" To ensure a fair comparison, we configure their working (short-term) memory to be the same as ours (w). Additionally, As evident from Tables g and 9 LSTR 44 consistently achieves relatively low performance, particularly with Edit scores of 5.0% and 4.9% 50Salads and Breakfast datasets, respectively. This suggests severe over-segmentation in their predictions. Moreover, these performances are inferior to those of our baseline model (casual TCN), indicating that direct adoption of online detection models for the segmentation task is not ideal.
adjust its long memory accordingly provide access the entire past sequence.
Amongst all datasets, Breakfast is the most challenging, with significant performance gap between offline and online models, particularly on segmental metrics. Notably, the FI @50 score on Breakfast experiences drastic drop of 4/5, from 47.5% to 8.3%, highlighting the difficulty of the online segmentation task with videos that are more complex. Nonetheless, we still achieve a modest absolute performance improvement of 2%. Furthermore, our post-processing technique, significantly boosts segmental performance, nearly tripling the original performance, albeit with slight decrease in Acc. This underscores the effectiveness of post-processing technique in mitigating the over- segmentation. MV-TAS [33 tackles online segmentation but under a multi-view setting. It leverages multi-view information and offline model as a reference for online segmentation. Despite this, even our baseline model, depicted in the third-to-last row of Table showcases a notable performance improvement (55.3% 41.6%) ) over MV-TAS KK] This considerable margin emphasizes the competitiveness of our baseline model.
When compared to offline models, semi-online inference with pesst-rocessing manages to surpass the offline model MS-TCN [IT] on 50Salads dataset across the segmental metrics and reaches around 90% of the accuracy of the best-performing DiffAct (23.. On Breakfast, our approach lags behind the offline model in both frame-wise accuracy and segmental metrics.
Qualitative Result. Fig. 3 qualitatively compares the segmentation results from different approaches. clear to see that LSTR suffers from the most prominent over-segmentation issue, which
short- teem- tent--tttt-ttt--freeeee

---

Breakfast
[8
Method
Acc
Edit
Fl @
(10, 25, 50)
offfine
MS-TCN
69.3
67.3
64.7
59.6
47.5
ASFormer
45
73.5
75.0
76.0
70.6
57.4
DiffAct
22
75.1
76.4
80.3
75.9
75.1
MV-TAS 13
41.6
LSTR 44
24.2
4.9
5.5
3.9
1.7
online
Causal TCN
55.3
18.7
15.1
11.7
8.3
Ourg online
56.7
19.3
16.8
13.9
9.3
Ourgtomine
p.p.
52.9
55.7
54.8
45.8
30.5
Ourssemi
57.4
19.6
17.8
14.8
10.1
Ours semi
+ P
53.8
57.5
56.4
47.3
31.4
Table 9: Comparison with the saate-oftthe-nn art methods on Breakfast.
remains significant after the post-processing. Under the same configuration, our semi-online achieves slightly better results compared to the frame-by-framy online inference. Our post-preee prot--rocessing, when applied, successfully removes the short fragments (blue boxes) in the raw prediction and refines the segmentation output. However, it may reduce accuracy, particularly at action boundaries (red boxes).
For failure cases, have the following due to the need for more frame information to predict new actions, especially when facing semantic ambiguities action boundaries; 2) Persistent over-segmentation happens when the network makes incorrect but confident predictions, which could be improved with a stronger backbone or better temporal context modeling.
observations: 1) Detection of action start often delays
LSTR 44
post proc.
Ours online
post proc.
Ours somi
post proc.
MS-TCN
GT
Figure 3: Visualization of segmentation outputs for sequence "rgh-01-1" from 50Salads 38
4.3 Runtime Analysis
We evaluate the runtime performance of approach using Nvidia A40 GPU with both pre- computed 13D features and raw streaming RGB inputs, and present the inference times Table As shown, our approach can achieve up 238. FPS when using pre-computed I3D features. To calculate the runtime for the entire segmentation pipeline, we take into consideration the computational overheads of optical flow calculation and the I3D feature extraction. By leveraging GPU backend for optical flow calculation, our full framework is able to achieve runtime of 33.8 FPS.
Inference Latency. The inference speed presented above is identical for both online and semi-online inference modes since their input sizes are same. However, the latency can differ. In the online mode, inference is performed on per-frame basis, meaning its latency is only dependent the

---

inference speed. In contrast, the semi-onling mode incurs additional latency as it requires gathering frames up the clip lengths before forming inputs.
Online inference offers hetter real-time responsiveness compared to semi-online inference, but the latter achieves superior performance modes depends on the application's 's priorities: if the real-time inference is critical, online inference is preferable; however, if accuracy more important and the task is less time-sensitive, semi-online inference is recommended.
discussed in Sec. [4.] The choice between these two
Ours (I3D)
OF Comp.
I3D Comp.
Ours (raw)
Time (ms)
4.2
4.8
20.5
29.5
FPS
238.1
208.3
48.8
33.8
Table 10: Runtime profile (in and FPS).
Limitation. In this work, we only evaluate our approach on cooking videos, however, handling diverse and real-world videos may present several additional challenges. One common scenario involves interrupted actions, where a subject abruptly switches to a different action, leaving the ongoing action unfinished. These interruptions can be challenging for the model to handle effectively, Additionally, the extended length of the video poses another challenge. Streaming videos can be infinitely long, effectively managing and preserving long-form history within fixed memory budget becomes a critical issue.
Conclusion
This paper presents the first framework for the online segmentation of actions in procedural videos. Specifically, we propose an adaptive memory bank designed to accumulate and condense temporal context, alongside a feature augmentation module capable of injecting context information into inputs and producing enhanced representations. In addition, we propose fast and effective pest-processing technique aimed at mitigating the ver-seementation problem. Extensive experiments on common benchmarks have shown the effectiveness of our approach in addressing the online segmentation task.
Acknowledgments and Disclosure of Funding
This research is supported by the National Research Foundation, Singapore under its NRF Fellowship for AI (NRF-NRFFAI FFAT1-00F 1-1191-0000)) Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not reflect the views of National Research Foundation, Singapore.
References
[1] E. Bahrami, G. Francesca, and J. Gall. How much temporal long-term context needed for action segmentation? In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 10351-10361, 2023.
[2] S. Cao, W. Luo, B. Wang, W. Zhang, and L. Ma. E2e-load: end-to-end long-form online action detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10422-10432, 2023.
[3] Carreira and A. Zisserman. Quo vadis, action recognition? new model and the kinetics dataset. In proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 6308, 2017.
[4] J. Chung, C. Gulcehre, K. Cho, and Bengio. Empirical evaluation of gated recurrent neural networks sequence modeling, arXiv preprint arXiv: 1412.355, 2014.
[5] R. De Geest, F. Gavves, A. Ghodrati, Z. Li, C. Snoek, and T. Tuytelaars. Online action detection. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11- 11-14, 2016, Proceedings, Part V 14, pages 269-284. Springer, 2016.
10

---

[6] G. Ding, S. Fadime, and A. Yao. Temporal action segmentation: An analysis of modern techniques. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
[7] G. Ding, H. Golong, and A. Yao. Coherent temporal synthesis for incremental action segmenta- tion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 28485-28494, 2024.
[8] G. Ding and A. Yao. Leveraging action affinity and continuity for semi-supervised temporal action segmentation. In European Conference Computer Vision, pages 17-32. Springer, 2022.
[9] G. Ding and A. Yao. Temporal action segmentation with high-level complex activity labels. IEEE Transactions on Multimedia, 25:1928-199 1939, 2022.
[10] Z. Du, X. Wang, G. Zhou, and Q Wang. Fast and unsupervised action boundary detection for action segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3323-3332, 2022.
[11] Y. A. Farha and Gall. Ms-ten: Multi-stage temporal convolutional network for action segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3575-3584, 2019.
[12] A. Fathi, X. Ren, and J. M. Rehg. Learning to recognize objects in egocentric activities. In CVPR 2011, pages 3281-3288. IEEE, 2011.
[13] R. Ghoddoosian, I. Dwivedi, N. Agarwal, C. Choi, and B. Dariush. Seakly-supervised online action segmentation multi-view instructional videos. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 13780-13790, 2022.
[14] M. Guermal, A. Ali, R. Dai, and F. Brémond. Joadaa: joint online action detection and action anticipation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 6889 6898, 2024.
[15] D.-A. Huang, 7. Yu, and A. Anandkumar. Minvis: A minimal video instance segmentation framework without video-based training. Advances in Neural Information Processing Systems, 35:31265-31277, 2022.
[16] H. Idrees, A. R. Zamir, Y.-G. Jiang. A. Gorban, Laptev, Sukthankar, and M. Shah. The thumos challenge on action recognition for videos "in the wild".. Computer Vision and Image Understanding, 155:1-23, 2017.
[17] Y. Kong and Fu. Human action recognition and prediction: survey. International Journal of Computer Vision 130(5):136611) 2022.
[18] H. Kuchne, A. Arslan, and T. Serre. The language of actions: Recovering the syntax and semantics of goal-directed human activities. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 780-787, 2014.
[19] S. Kumar, S. Haresh, A. Ahmed, A. Konin, M. Z. Zia, and Q.-H. Tran. Unsupervised action segmentation by joint representation learning and online clustering. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 201 20-851 2022.
[20] C. Lea, M. D. Flynn, R. Vidal, A. Reiter, and G. D. Hager. Temporal convolutional networks for action segmentation and detection. In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 156-165, 2017.
[21] J. Li, P. Lei, and S. Todorovic. Weakly supervised energy cnsegy-based learning for action segmentation. In Proceedings of the IEEE/CVF international conference on computer vision, pages 2243-6251. 2019.
[22] J. Li and S. Todorovie. Action shuffle alternating learning for unsupervised action segmentation. In Proceedings of the IEFE//VV Conference on Computer Vision and Pattern Recognition, pages
2021.
11

---

[23] S. Li, Y. A. Farha, Y. Liu, M.-M. Cheng, and J. Gall. Ms-ten++: Multi-stage temporal convolutional network for action segmentation. IEEE transactions on pattern analysis and machine intelligence, 45(6):6647-66588 2020.
[24] S. Lin, L. Yang, I. Saleemi, and S. Sengupta. Robust high-resolution video matting with temporal guidance. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 238-247, 2022.
[25] D. Liu, Q. Li, A.-D. Dinh, T. Jiang, M. Shah, and C. Xu. Diffusion action segmentation. In International Conference Computer Vision (ICCV), 2023.
[26] Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEF/CVF international conference computer vision, pages 10012-10022, 2021.
[27] D. Moltisanti, S. Fidler, and D. Damen. Action recognition from single timestamp supervision in untrimmed videos. In Proceedings of the IEEE/CVE Conference on Computer Vision and Puttern Recognition, pages 9915-9924, 2019.
[28] M. K. Myers, N. Wright, A. S. McGough, and N. Martin. O-tale: Steps towards combating oversegmentation within online action segmentation. arXiv preprint arXiv:2404, 06894, 2024.
[29] R. Rahaman, D. Singhania, A. Thiery, and A. Yao. A generalized and robust framework for timestamp supervision in temporal action segmentation. In European Conference on Computer Vision, pages 279-296. Springer, 2022.
[30] A. Richard, H. Kuehne, and J. Gall. Action sets: Weakly supervised action segmentation without ordering constraints. In Proceedings of the IEEE conference on Computer Vision and Patter Recognition, pages 5987-5996, 2018
[31] A. Richard, H. Kuehne, A. Iqbal, and J. Gall. Neuralnetworr--ieerhi: A framework for weakly supervised video learning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 7386-7397 2018
[32] S. Sarfraz, N. Murray, V. Sharma, A. Diba, L. Van Gool, and R. Stiefelhagen. Temporally- weighted hierarchical clustering for unsupervised action segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11225-11234, 2021.
[33] F. Sener, D. Singhania, and A. Yao. Temporal aggregate representations for long-range video understanding. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XVI 16, pages 154-171. Springer, 2020.
[34] F. Sener and A. Yao. Unsupervised learning and segmentation of complex activities from video. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8376, 2018
[35] 7. Shou, D. Wang, and S.-F. Chang. Temporal action localization in untrimmed videos via multi-stage cnns. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 1049-1058, 2016.
[36] D. Singhania, R. Rahaman, and A. Yao. Iterative coasrast-classify for cemi-supervised tem- poral action segmentation. Proceedings of the AAAI Conference on Artificial Intelligence, 36(2):2262-223) Jul 2022.
[37] D. Singhania, R. Rahaman, and A. Yao. C2f-ten: A framework for semi-and fully-supervised temporal action segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
[38] S. Stein and S. J. McKenna. Combining embedded accelerometers with computer vision for recognizing food preparation activitics. In Proceedings of the 2013 ACM international joint conference Pervasive and ubiquitoos computing, pages 729-738, 2013.
[39] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, t.. Kaiser, and
I. Polosukhin. Attention is all you need. Advances neural information processing systems, 30, 2017.
12

---

[40] S. Venkatesh, D. Moffat, and E. R. Miranda. Investigating the effects of training set synthesis for audio segmentation of radio broadcast. Electronics, 10(7):827, 2021.
[41] Wang, G. Chen, Huang, L. Wang, and T. Lu. Nemory-and-anticipatinn transformer for online action understanding. In Proceedings of the IEEE/CVE International Conference Computer Vision, pages 9824-13835, 2023.
[42] X. Wang, S. Zhang, Z. Qing, Shao, Z. Zuo, C. Gao, and N. Sang. Oadtr: Online action detection with transformers. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 7565-7575, 2021.
[43] M. Xu, M. Gao, Y.-T. Chen, L. S. Davis, and D. J. Crandall. Temporal recurrent networks for online action detection. Proceedings of the IEEE/CVF international conference on computer vision, pages
2019.
[44] M. Xu, Y. Xiong, H. Chen, X. Li, W. Xia, Z. Tu, and S. Soatto. Long short-term transformer for online action detection. Advances Neural Information Processing Systems, 34:1086-1099 2021.
[45] F. Yi, H. Wen, and T. Jiang. Asformer: Transformer for action segmentation. In BMVC, 2021.
[46] K. Ying, Q. Zhong, W. Mao, Z. Wang, H. Chen, L. Y. Wu, Y. Liu, C. Fan, Y. Zhuge, and C. Shen. Ctvis: Consistent training for online video instance segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 899-908, 2023.
[47] Zhang, X. Tian, Y. Wu, S. Ji, X. Wang, Y. Zhang, and Wan. Dvis: Decoupled video instance segmentation framework. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1282-122 1291, 2023.
[48] H. Zhao, A. Torralba, L. Torresani, and Z. Yan. Hacs: Human action clips and segments dataset for recognition and temporal localization. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pages 8668-8678, 2019.
13

---

A Appendix supplemental material
A.1 Standard vs. Causal Convolution
Standard convolution. As shown in Fig. 4 (left), the receptive field of a standard convolution includes both past and future inputs. Mathematically, for input sequence sr()) and filter h(k), the output (t) at time f is given by:
K x(t k)
v(1) = Σ
h(k)
(8)
k=-K
where K is the size of the filter. This means the output time depends inputs from K to + K.
Causal convolution. As shown in Fig. 4 (right), the receptive field of causal convolution includes only the past and current inputs, ensuring that the output at time does not depend on future inputs. Mathematically, the output y(t) is given by:
K
vlt)
Σ
h(k)
art k)
(9)
k=
where K is the size of the filter. This means the output time depends only on inputs from to t -- K.
A.2 CFA Formula
The attention mechanism
39 is written
Q x KT x V
Attention(,, K) SoftMax
(10)
Vd
where Q, K, V represents query, key and value, respectively, and d is the hidden dimension.
We use Transformer decoder 39 as follows:
obtain the memory encoding, and TransDecoder)) is formulated
M --1
MM- M,- CrossAttm (MatttmMMM,,-0- Ck
GRU GRRU FFN
Ck
(11)
Here, the FFN Freed-Forwaad Network) is two-layer fully connected network, M₁-1 the memory bank, which first undergoes self-attention. The output of the self-attention mechanism is used as the query cross-attention, where c effective memory encoding M
,RRU serves as the key and value. This interaction results more
T We split the input feature of size C' x T' to windows with size C' x 2. and perform Swin 26 self attention within each local window independently, and Cross attention for every consecutive local windows. The Swin attention mechanism can be formulated as:
(QXK) +B
KT +B)
Swin Attention Q, K, V) = SoftMax
+B)XV
XV
(12)
Va
Where the B is the relative position of the window. Then, method produces context-augmented features e using Eq. 3 and Eq. (4)
Our colf-attention based on Eq. 12
GRU CR
GRUUST x (c
SelfAutn (CE
GRU GRU GRU
SoftMax
+ +B)
x CE
GRU
(13)
Vd
where e as the clip features and & as the current step, Gj passes through a GRU to obtain CGRU Next, we use cross-attention interact with the output the self-attention SA with the output memory bank of the Transformer decoder, MTD
(MTT
CrossAtt SA AD
M = SoftMax
c
+ B x TDD
(14)
Vd
14

---

Standard Convolution
Causal Convolution
Figure Standard vs. Causal Convolution
A.3 Implementation
Hyper-parameters. As shown in Table 11 the for each dataset. GTEA uses shorter window size because the longest video is only about 2000 frames, whereas the other datasets all use a window size of 128. The optimal confidence threshold for segmental metrics varies for each dataset: for GTEA 0.6; for 50Salads is 0.9; and for Breakfast 0.8.
settings are generally the same
Hyper-parameters
GTEA
50Salads
Breakfast
Learning rate
5e
5c
5e
Epochs
50
50
50
No. GRU layers
1
64
128
128
σ
1/16
1/16
1/16
0
0.6
0.9
0.8
iteration I
2
2
TD heads
8
SwinAttn heads
4
4
No. Causal TCN stages
1
No. Causal dilated Conv layers
10
10
10
Table 1I: Hlyper-parameter settings for GTEA, 50Salads, and Breakfast datasets.
A.4 AsFormer Performance
We conduct experiments three common TAS datasets, where replace the MS-TCN backbone with AsFormer. MS-TCN, the transition to an online method is relatively straightforward, as only requires replacing all the standard convolution layers with causal convolution layers. However, in AsFormer, the transformation involves more extensive modifications. the convolution layers with causal convolutions, we also modify the standard attention layers into causal attention layers. Furthermore, we incorporate our proposed GRU, CFA, Memory Bank, and Post- procespoc module to ensure that AsFormer transitions from offline method to online method. Our approach remains highly effective boosting online segmentation performance while maintaining the strengths of the AsFormer architecture.
addition to replacing
15

---

GTEA
50Salads
Brcakfast
p.p.
Acc
Edit
FI
[10,
25, 50)
Ace
Fdit
FI
@
{10,
25, 50}
Acc
Edit
Fl
@
{10, 25, 50)
Offline
79.7
84.6
90.1
88.8
79.2
85.6
79.6
85.1
83.4
76.0
73.5
75.0
76.0
70.6
57.4
Online
75.0
69.7
77.7
74.0
62.0
77.5
29.1
37.9
35.3
28.6
64.9
32.1
31.2
27.4
20.2
72.8
77.8
84.5
80.8
64.2
69.4
36.4
70.6
65.7
52.3
63.1
60.1
61.6
54.3
39.2
76.5
71.3
79.0
76.7
63.1
78.5
29.7
38.5
36.2
30.4
64.8
37.0
33,9
30.0
22.6
Semi
74.7
79.6
86.3
82.8
67.0
71.0
64,9
72.2
67.2
53.8
64.0
63.2
64.9
57.5
43.0
Table 12: Performance of our approach when using ASFormer as the segmentation backbone.
NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Please refer to Sec.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Please refer to Sec.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they seale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
16

---

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: Please refer Sec.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proof's provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Answer: [Yes]
Justification: Please refer to Sec.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
17

---

(c) If the contribution is a new model (e.g.. a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described supplemental material?
Answer:[NAl
Justification:
The code will made available once accepted.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. ec/
public/guide httii//ggiidos/ooouuubiisssssss "oodeSumiissionee
aasionPoliyy ollicyy for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https for more details.
/ htips.cccpublic//eee Setggidddde//eeSubiemeaeaa /wtp///ooeeSeseeeeeeeeeeee
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: Please refer Sec.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [NA]
Justification: Our method uses fixed seed, so there is error bar due randomness.
18

---

Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the eπror bar is the standard deviation or the standard error of the mean.
It is OK to report I-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out of range (c.g. negative error rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
Question:
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [NA]
Justification: single NVIDIA RTX 3090 GPU is sufficient for training.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
Our code meets the NeurIPS Code of Ethics requirements.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [NA]
19

---

Justification:
Our work performs no societal impact.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (c.g., pretrained language models, image generators, scraped datasets)?
Answer: [NA]
Justification: Our paper poses no risk for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Please refer Sec.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
20

---

The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the palaaets.ppperssttt paperssii thcode. com con/datases package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: Our paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
21

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
22


Reviews:
[{"review_id": "29N6sT0lDT", "reviewer": "Reviewer_xPL1", "summary": "This paper presents the first online framework for temporal action segmentation. At the core of the framework is an adaptive memory designed to accommodate dynamic changes in context over time, alongside a feature augmentation module that enhances the frames with the memory. A post-processing approach is proposed to mitigate the severe over-segmentation in the online setting. The method achieves state-of-the-art performance on three common segmentation benchmarks.", "questions": "As an online paradigm, why use offline models as baselines instead of building online models from scratch? The solution of replacing standard convolutions with causal convolutions is not very elegant.", "limitations": "see weakness and questions.", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well written and easy to be followed.\n2. Exploring online paradigms in temporal action segmentation is meaningful.", "weaknesses": "1. Novelty is a big issue. The proposed adaptive memory bank mechanism has been explored in the video object segmentation tasks[1]. Also, semi-online inference scheme has been proposed in video instance segmentation tasks[3].\n[1] Video Object Segmentation with Dynamic Memory Networks and Adaptive Object Alignment, iccv\n[2]Video Object Segmentation with Adaptive Feature Bank and Uncertain-Region Refinement,neurips\n[3] A Generalized Framework for Video Instance Segmentation, cvpr\n\n2. The context-aware feature augmentation (CFA) module mainly consists of self-attention and cross-attention, adaptive memory bank was widely used in video detection and segmentation tasks, so the proposed module is not novel enough.\n\n3. Is the Trans. Decoder necessary in the CFA module? For example, directly using memory as K and V of CA or performing SA together with ck.\n\n4. In semi-online inference, why use non-overlapping clips? Would using overlapping clips for sampling and voting classification yield better results? \n\n5. The performance improvement of Edit and F1 brought by post-processing is so amazing, why is there such a big improvement but Acc has decreased.\n\n6. Compared to the latest offline methods, there is still a significant gap in the performance of online methods. The online method has advantages in terms of training cost and inference speed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for acknowledging the task importance and our effective presentation of the work.\n\n**Weaknesses** \n\n----\n\n**W1&2. Novelty on memory bank, semi-online inference, self- and cross-attention**\n\nThanks for your comments.  Indeed, attention and memory banks are common architectural components and semi-online inference is also an established concept.  Note that we don’t claim the memory bank, attention mechanisms and semi-online inference as our technical contributions.  \n\nThe distinction and novelty lie in how the objective and design are tailored to the specific task rather than the architectural components or abstract concepts stand-alone.  Our focus is centered on the unique integration of the memory bank, attention mechanism, and semi-online inference for the online TAS task. \n\nHere are detailed distinctions:\n\n**Memory bank**\n\nIn [1,2], the memory design supports instance matching where each memory token represents object instances and is selectively updated via feature similarities [2]. While our memory design accumulates contextual information for feature enhancement, storing contexts at different granularities: clip-wise coarse long-term memory and frame-wise fine-grained short-term memory. Our memory capacity adapts over time to retain as much contextual information as possible. \n \n[1,2] update Q directly based on K and V, whereas ours retrieves relevant memory for the query clip with Trans. Decoder and then enhance features with SA and CA.\n\nIn addition, our method shows over a 30% increase in Acc (Table 9) compared to existing work like LSTR [44], using similar attention and memory techniques.\n\n**Semi-online Inference**\n\nThank you for pointing out [3], semi-online inference is not a claimed contribution but is presented to offer a more comprehensive evaluation due to the clip-based nature of the model's inputs and outputs. \n\nIn summary, as stated in L43-49, our technical contributions are: \n\n1) Addressing the novel online TAS problem;\n\n2) Developing CFA leveraging adaptive memory;\n\n3) A fast and effective post-processing for over-segmentation in the online setup.\n\n----\n\n**W3. Trans. Decoder necessity**\n\n**A**: We include a Trans. Decoder is to gather context information specific to the current clip, which we then use to enhance the clip features for better online segmentation. \n\n| Trans Dec| Mem|Acc|Edit|F1@10| F1@25  | F1@50 |\n|-|-|-|-|-|-|-|\n| w/ | w/  | 82.4  | 32.8  | 43.0    | 41.1   | 34.7   |\n| w/o | w/ | 80.6  | 29.1  | 39.5    | 36.2   | 29.3   |\n| w/o | w/o | 77.3  | 28.3  | 35.9    | 33.6   | 23.1   |\n\nThe ablation results above (on 50Salads) indicate that performance decreases when the component is removed and memory is directly fed as K and V for CA (row 2). Further removal of the memory (row 3) leads to an even greater performance drop, highlighting the importance of the context information in the online task.\n\n----\n\n**W4. Overlap semi-online performance**\n\n**A**: The table below shows that with a stride of $\\delta=64$ and a window size of $\\omega=128$, overlaps lead to better results than the frame-wise online mode but lag behind the non-overlapping mode.\n\n| Method  | Voting | Acc   | Edit  | F1 10   | F1 25  | F1 50  |\n|-|-|-|-|-|-|-|\n| Semi-online  | no | 82.4  | 32.8  | 43.0    | 41.1   | 34.7   |\n| ½ overlap  | yes | 81.1  | 29.5  | 40.4    | 39.3   | 30.1   |\n| online  | no  | 79.1  | 29.0  | 38.5    | 35.5   | 28.3   |\n\nOverlapping results are affected by multiple predictions with various contexts, while the non-overlapping case benefits from a shared context, leading to better accuracy and less over-segmentation.\n\n----\n\n**W5. Acc decreases after post-processing**\n\n**A**: Segmental metrics and frame-wise accuracy are not well-aligned in TAS. A high accuracy doesn’t necessarily ensure less over-segmentation and vice versa.  Consider the extreme case where every other frame is interspersed with the wrong action, the segmental measures are very poor, but the accuracy is still 50%. This trade-off is also observed in [R1].\n\nOur post-processing prioritizes reducing over-segmentation over frame-wise accuracy. **Fig. RA in the rebuttal pdf** (sample example as Fig. 3) shows post-processing: 1) removes fragments (black boxes). 2) may reduce Acc, particularly at action boundaries (red boxes).\n\n[R1] Unified Fully and Timestamp Supervised Temporal Action Segmentation via Sequence to Sequence Translation, ECCV 2022.\n\n----\n\n**W6. Performance gap between online and offline**\n\n**A**: We respectfully disagree that a performance gap between online and offline models is a weakness. \n\nSuch a gap is well-expected and is directly attributed to the online nature, in which future temporal context is not available for making predictions.  In a similar fashion, [R2] found that under the same offline setup, performance drops when the input window size is reduced and cannot account for sufficient context.\n\n[R2]. How much temporal long-term context is needed for action segmentation? ICCV 2023. \n\n**Questions**\n\n----\n\n**Q1. Why use offline models?**\n\n**A**:  We thank the reviewer for the question.  \n\nSuccessfully solving TAS requires merging visual features over long temporal ranges of context, while preserving local semantics.  Existing offline models are designed specifically for this task and excel at it, making them a natural starting point rather than building from scratch.  \n\nAdditional advantages include: 1) using the same backbone to facilitate performance evaluations and comparisons to the offline model and 2) analyzing specific performance discrepancies to identify meaningful areas for improvement.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yfjPpgTlUS", "reviewer": "Reviewer_itXK", "summary": "This paper introduces OnlineTAS, the first fully-supervised online framework for temporal action segmentation (TAS). The main contributions include:\n1. A context-aware feature augmentation (CFA) module that incorporates an adaptive memory to enhance frame representations with temporal context.\n2. An adaptive memory bank that accumulates short-term and long-term context information.\n3. A post-processing technique for online boundary adjustment to mitigate over-segmentation.\n4. State-of-the-art performance on three TAS benchmarks in an online setting.", "questions": "The following questions are mostly related to weaknesses directly or indirectly:\n\n1. Have you explored the impact of different feature extractors and different backbones on the performance of OnlineTAS?\n\n2. Can you elaborate on how OnlineTAS might be adapted for weakly-supervised or unsupervised settings?\n\n3. How sensitive is the method to the choice of hyperparameters, particularly the clip size and memory length?\n\n4. Your post-processing technique seems effective in mitigating over-segmentation. Have you explored how this technique might be integrated into the model itself during training, rather than as a post-processing step?\n\n5. The paper introduces both online and semi-online inference modes. How do these two modes compare in terms of latency and accuracy trade-offs? In what scenarios would one be preferable over the other?", "limitations": "The authors acknowledge some limitations, such as the computational intensity of the offline segmentation model used as a backbone and the reliance on pre-extracted features. These factors could hinder real-time application of the framework. However, the discussion of limitations could be expanded to include potential challenges in adapting the method to more diverse datasets or real-world streaming scenarios.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The CFA module and adaptive memory bank provide an innovative approach to incorporating temporal context in an online setting.\n2. The proposed post-processing technique effectively mitigates the over-segmentation problem common in online TAS.\n3. The authors demonstrate the effectiveness of their approach on three standard TAS benchmarks, showing consistent improvements over baselines.", "weaknesses": "1. The paper lacks a thorough discussion of the computational complexity and resource requirements of OnlineTAS. For an online method intended for real-time applications, this is a critical aspect. The authors mention that their approach uses a \"single-stage TCN\" for efficiency, but don't provide concrete details on inference time, memory usage, or how these scale with video length. A comparison of computational requirements with offline methods and other online video understanding tasks would be beneficial. This analysis should include considerations for both the online and semi-online inference modes.\n\n2. The paper presents quantitative results but lacks a qualitative error analysis. A detailed examination of failure cases could provide valuable insights into the limitations of the approach and guide future improvements. For instance, analyzing scenarios where OnlineTAS performs poorly compared to offline methods or where over-segmentation persists despite post-processing could be informative.\n\n3. While the paper introduces an adaptive memory bank, there's limited discussion on how this approach scales to very long videos or continuous video streams. It's unclear how the method would perform in scenarios where the video length greatly exceeds the memory capacity, which is a likely scenario in many real-world applications.\n\n4. For an online method, real-time performance is crucial. However, the paper doesn't provide metrics such as frames per second processing speed or end-to-end latency on standard hardware. This information is essential for assessing the practical applicability of the method in real-time scenarios.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for recognizing the novelty of our CFA module for the online TAS problem and the effectiveness of the post-processing for mitigating the over-segmentation issue.\n\n**Weakness**\n\n----\n\n**W1&4. Runtime analysis and computational requirements**\n\n**A**: We thank the reviewer for the comment.\n\n**Runtime**\n\nKindly refer to our response to W1 for Reviewer aGP1.\n\n**Computation requirement**\n\n| |GPU Mem Requirement |\n|-|-|\n| MSTCN| 551M (10k frames)|\n| ASFormer| 716M (10k frames)|\n| Ours|1.2G (fixed) |\n\nWe compare GPU memory requirements for inference between offline and online models. Offline models like MSTCN and ASFormer use 551M and 716M per every 10,000 frames, respectively. Note that the memory requirement will scale linearly with the actual video length as these models operate on the full sequence. In contrast, our model has a fixed memory requirement of 1.2G for each inference, regardless of the actual video length. This facilitates its application in dealing with online streaming videos that can be infinitely long.\n\n----\n\n**W2. Error analysis.**\n\n**A**: Thanks for the suggestion. We show some qualitative examples in Fig. 3, and we will add more to the revision. For failure cases, we have the following two observations:\n\n1) Action start often delays due to the need for more frame information to predict new actions, especially when facing semantic ambiguities at action boundaries.\n\n2) Persistent over-segmentation happens when the network makes incorrect but confident predictions, which could be improved with a stronger backbone or better temporal context modeling.\n\n----\n\n**W3. How memory scales with video lengths**\n\n**A**: We thank the reviewer for the question. When the memory capacity is exceeded, our method discards the earliest memory in a FIFO manner. \n\nOur ablation study (Table 5) includes scenarios where the video length exceeds the memory limit. For memory sizes of 16, 32, and 64, the earliest memory are discarded as the average length of 50Salads is ~5.8K frames. With the size reducing, the performance gradually drops and reaches the lowest of 79.8 in Acc compared to the peak of 82.4. Note that with the memory size set to 16, our approach only retains long-term information from up to 192 frames, 30x less than the average video length. \n\n----\n\n**Questions**\n\n**Q1. Different feature extractor and backbone**\n\n**A**:  Thanks for the suggestion.  We experiment on three common TAS datasets replacing the MS-TCN backbone with ASFormer and report the results in **Table RA in the rebuttal pdf**. Our approach is still effective in boosting online segmentation performance. \n\nUnfortunately, we weren’t able to conduct the experiments with other optional feature extraction backbones due to the time and resource constraints for the rebuttal. We will add this to the camera-ready.\n\n**Q2. Online setup under other forms of supervision**\n\n**A**: Thanks for the interesting question. Weakly- and unsupervised approaches typically involve iterative model training and label estimation. Our post-processing technique, as a label refinement process, may be used in such setups to refine network prediction and generate temporally consistent pseudo-labels. The two lines of work are orthogonal and serve as interesting thoughts for future work of addressing the online problem. \n\n----\n\n**Q3. Hyperparameter sensitivity**\n\n**A**: These studies are already reported in Table 5.  Our approach is reasonably tolerant to the clip size and memory length change. For example, the performance only drops 1% when the size is reduced to 1/4 (32) of the original default (128). \n\n----\n\n**Q4. Integrating post-processing for training**\n\n**A**: That is a good question. The post-processing is to refine the frame labels predicted by the model; we can see its application in weakly and unsupervised setups to refine pseudo labels as we explained in response to W2. However, as our current approach is fully supervised, there’s no need for label correction during training as the ground-truth labels are provided. \n\n----\n\n**Q5. Latency accuracy trade-off analysis between online and semi-online modes**\n\n**A**: The inference speed for one pass is identical for both modes as their input sizes are the same. However, their latency is different. Online inference operates on a frame basis and the latency is dependent only on the inference speed while semi-online mode requires additional latency to gather frames up to the clip lengths (128 frames at a standard 25 FPS corresponds to 5.12 seconds).\n \nIn terms of performance, the online approach is less competitive than the semi-online with an average gap of around 2-5% (see Table 1). This is likely because of the better preservation of temporal continuity in the semi-online setup,  which we discussed in L.219-222. \n \nTo summarize, online inference has a better real-time response than semi-online inference. Semi-online inference achieves better performance. The choice of these two modes will depend on the application priority, if the real-time inference is required, online would be preferred, if accuracy is preferred and the task is less time-sensitive, then semi-online is suggested.\n \nWe thank the reviewer for the constructive question and will include these discussions in our updated manuscript. \n\n----\n\n**Limitations**\n\n**L1. Potential challenges in diverse and real-world streaming cases**\n\n**A**: Thank you for your suggestion. Handling diverse and real-world videos presents several challenges. \n\nOne common scenario involves interrupted actions, where a subject abruptly switches to a different action, leaving the ongoing action unfinished. These interruptions can be challenging for the model to handle effectively.\n\nAdditionally, the extended length of the video poses another challenge. Streaming videos can be infinitely long, so effectively managing and preserving long-form history within a fixed memory budget becomes a critical issue.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ESCHY3cHZr", "reviewer": "Reviewer_aGP1", "summary": "This paper proposes an online baseline for temporal action segmentation. The method is built upon causal TCN and integrates a GRU, attention-based feature aggregation, and a memory bank. A heuristic online post-processing method is also proposed.", "questions": "NA", "limitations": "The limitation has been discussed regarding the real-time inference. This is good.", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "1. I like the statement claiming the contribution as a baseline for action segmentation. It is a modest and accurate claim. The paper provides a reasonable online baseline for this task. Online action segmentation is also a meaningful task.\n\n2. The proposed post-processing method is intuitive and effective.\n\n3. The experiments and the ablation studies show good results.", "weaknesses": "1. The major concern is about whether the method can achieve real-time inference speed. This is crucial for the online setting. Currently, no inference speed is reported, for example, the FPS including the feature extraction. Although the authors discussed this in the limitation section, this is a major weakness, given that the method is proposed as a baseline for the online setting.\n\n2. Experiments on new benchmarks such as Assembly101 would be interesting.\n\n3. I would recommend the authors remove the statement \"we are the first to establish a fully supervised online framework for TAS\". There are some concurrent works that became available before the NeurIPS submission deadline or slightly after that, e.g., O-TALC and Progress-Aware Online Action Segmentation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for finding our contribution claims to the meaningful online TAS task modest and accurate. We address the reviewer's comments below.\n\n**Weaknesses**\n\n----\n\n**W1. Inference speed**\n\n**A**: Runtime evaluation, conducted on our Intel Xeon Gold 6442Y (2.6 GHz) and a single Nvidia A40 GPU, is reported in time (ms) per frame / FPS in the table below. Note that the inference speed is identical for online and semi-online modes due to their identical input length. \n\n| Optical Flow Algorithm | Optical Flow Calculation  | I3D - RGB  | I3D - Optical Flow | Ours (I3D input)  | Ours (raw RGB input)  |\n|-|-|-|-|-|-|\n|   | Time (ms) / FPS  | Time (ms) / FPS | Time (ms) / FPS | Time (ms) / FPS  | Time (ms) / FPS |\n| TV-L1 | 166.5 / 6   | 9.3 / 107  | 11.2 / 89 | 4.2 / 238 | 191.2 / 5 |\n| NVOFA$^*$ | 1.4 / 714   | 9.3 / 107  | 11.2 / 89 | 4.2 / 238  | 26.1 / 38  |\n\n[*]: NVOFA SDK. https://developer.nvidia.com/opticalflow-sdk\n\nThese are our findings:\n\n1) Our architecture achieves an inference speed of 238 FPS with precomputed I3D features.\n\n2) Feature extractions for both RGB and optical flow inputs are fast, reaching at least 3x the real-time rate of 25 FPS.\n\n3) The TV-L1 algorithm used by I3D, not optimized for online requirements, supports 6 FPS. However, Nvidia Optical Flow SDK (NVOFA) with hardware acceleration can boost this speed to 714 FPS.\n\nIn conclusion, to use I3D features, the optical flow calculation is the main bottleneck. Real-time application is feasible with more advanced acceleration algorithms.\n\n----\n**W2. Results on Assembly101**\n\n**A**: Due to the limited time, we could only experiment with a subset of a single view (C10095).  Results are shown in the table below. Our approach can also effectively achieve reasonable performance compared to the offline setup. The pure online for the Assembly101 dataset is very challenging in terms of segmental metrics (row 2), while our proposed post-processing significantly boosts segmental performance.  \n\n\n| Method        | Acc   | Edit  | F1 10   | F1 25  | F1 50  |\n|-|-|-|-|-|-|\n| Offline       | 37.1  | 30.7  | 31.6    | 27.8   | 20.6   |\n| Online        | 36.8  | 10.9  | 8.7     | 6.4    | 4.0    |\n| Online + p.p.  | 33.5  | 28.2  | 29.1    | 25.4   | 17.6   |\n\n\n----\n\n**W3. Contribution claim**\n\n**A**: Thanks for pointing out these concurrent works. After a close examination, O-TALC was released on Arxiv in mid-April indicating acceptance as a short and unindexed paper in TAHIR; while Progress-Aware Online Action Segmentation only became available at CVPR one month after our submission. We thank the reviewer and will revise our statement in our updated manuscript.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all reviwers for their constructive comments and address each of their concerns in separate rebuttals.  The attched global rebuttal file contains 

1) 1 figure (Fig. RA) in response to Reviewer xPL1's W5 regarding the segment-accuracy tradeoff in post-processing.

2) 1 table (Table RA) in response to Reviwer itXK's Q1 regarding switching the backbone (ASFormer) for online TAS.

Meta-review: The paper received 3 borderline rejects during the initial review period. While the reviewers appreciated the problem setup and the clarity of presentation, concerns were raised about the novelty, limited error analysis, computational overhead for truly online segmentation, the heavy reliance on post-processing for reducing over-segmentation, and the performance gap between offline and online models. The author's rebuttal provided a response that addresses many of the concerns. While some concerns exist, the strengths outweigh the weaknesses, and I recommend acceptance since the proposed model will serve as a convincing baseline for online TAS. The authors are encouraged to include the details from the rebuttal in their final version.
Decision: Accept (poster)