Efficient Temporal Action Segmentation via Boundary-aware Query Voting
Authors: Peiyao Wang
Yuewei Lin
Erik Blasch
Jie Wei
Haibin Ling
Venue: NeurIPS 2024 poster (2024)
Keywords: efficiency
action understanding
action segmentation
action detection
Primary area: machine_vision

Abstract: Although the performance of Temporal Action Segmentation (TAS) has been improved in recent years, achieving promising results often comes with a high computational cost due to dense inputs, complex model structures, and resource-intensive post-processing requirements. To improve the efficiency while keeping the high performance, we present a novel perspective centered on per-segment classification. By harnessing the capabilities of Transformers, we tokenize each video segment as an instance token, endowed with intrinsic instance segmentation. To realize efficient action segmentation, we introduce BaFormer, a boundary-aware Transformer network. It employs instance queries for instance segmentation and a global query for class-agnostic boundary prediction, yielding continuous segment proposals. During inference, BaFormer employs a simple yet effective voting strategy to classify boundary-wise segments based on instance segmentation. Remarkably, as a single-stage approach, BaFormer significantly reduces the computational costs, utilizing only 6% of the running time compared to the state-of-the-art method DiffAct, while producing better or comparable accuracy over several popular benchmarks. The code for this project is publicly available at  https://github.com/peiyao-w/BaFormer.

Efficient Temporal Action Segmentation via Boundary-aware Query Voting
Peiyao Wang'
Yuewei Lin?
Erik Blasch
Jie Wei
Haibin Ling'
2 Stony Brook University, Brookhaven National Laboratory, 3 Air Force Research Laboratory, "The City College of New York {peiyaowang, hling stonybrook. edu, yslin@bnl... gov, erik. blasch@gmail. com, jwei@ccny cuny. edu
Abstract
Although the performance of Temporal Action Segmentation (TAS) has been improved in recent years, achieving promising results often comes with a high computational cost due to dense inputs, complex model structures, and resource- intensive post-processing requirements. To improve the efficiency while keeping the high performance, we present novel perspective centered on per-segment classification. By hameessing the capabilities of Transformers, we tokenize each instance token, endowed with intrinsic instance segmentation. To realize efficient action segmentation, we introduce BaFormer, a boundary- aware Transformer network. It employs instance queries for instance segmentation and global query for class-agnostic boundary prediction, yielding continuous segment proposals. During inference, BaFormer employs simple yet effective voting strategy to classify boundary-we segments based on instance segmenta- tion. Remarkably, single-stage approach, BaFormer significantly reduces the computational costs, utilizing only ~~6 % of the running time compared to the state-of-the-art method DiffAct, while producing better or comparable accuracy over several popular benchmarks. The code for this project is publicly available at htttps://github www.co/./ee... com/peiyao- #/BaFormer
video segment
Introduction
Temporal Action Segmentation (TAS) 28
notable endeavor in the study of untrimmed videos, aims to allocate an action label to each frame, enabling the detailed analysis of complex activities by identifying specific actions within long-form videos. It has extensive applications surveillance [1 12). sports analytics [00]. and human-computer interaction (38). Despite its critical importance, existing TAS models often grapple with high computational costs and lengthy inference times, limiting their applicability in real-time and resource-constrainee scenarios.
43
30 29 21 32).
The inefficiency of TAS arises from two primary factors: Long-form Input: Untrimmed videos in TAS often include tens of thousands of frames containing multiple action segments that ideally should be sparse in terms of action semantics. However, most approaches [15] 32 21 43) maintain frame-wise wise processing throughout the model to preserve dense information, aiming to enhance frame- wise prediction accuracy yet the cost of efficiency. Conversely, the straightforward downsampling method (88 fails to achieve the desired accuracy for dense predictions. 2) Heavy Model Structure: The adoption of multi-stage refinement models and various time-consuming post- processnnsssing tech- niques further contribute to the slow runtime of TAS models. For instance, although a previous
Coorespooniing authors
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

65
LiCodiett
DTL
06-
Belvmnerr
AStommer
Instance queries
a global query
ASRF
HASK
UVAST-FIIA
query masks
query classes
boundaries
SSTTA
MSTCN
iisttacee segmentation
Voting
DTGRM
UVAST UVAST+ valigament
dage Propeconiig
00
0.2
0.3
Iatxne Nadeo
L75
200
225
results
Figure 1: Accuray vs. inference time 50Salads. The bubble size represents the FLOPs in inference. Under different backbones, BaFormer enjoys the benefit of boundary-awar query voting with less running time and improved accuracy.
Figure 2: An efficient pipeline developed by our proposed BaFormer.
method 33 employs a single-stage model with transcripts to mitigate the heavy model structure, the algorithms used during post-processing remain time-intensive, as shown in Fig.
Our objective to transform the long-form video into a sparse representation, thereby reducing the temporal dimension processed by the model. Additionally, aim to employ a single-stage model coupled with an appropriate post-processing method to minimize the running time. Benefiting from the query-based model, which allows information to be compressed into limited number of queries, segments can be effectively represented as queries. By enabling each query to predict the corresponding class, start, and end, action segmentation can be seamlessly reframed as an action detection task. However, this significant compression of the long-form input adversely impacts the accuracy of dense predictions, as noted in 33. To address the limitation, we draw inspiration from recent advances in mask classification based image segmentation, particularly MaskFormer [0]. In TAS, rather than predicting frame-wiss results directly, we propose using set of queries to represent segments by decoupling their binary masks and their classes. The decoupling approach effectively conveys sparse information through limited queries, while preserving dense information via binary masks. Furthermore, the continuous nature of features in the temporal dimension introduces discontinuities in learning frame-wise masks. To address the issue discontinuity, we introduce an additional query to learn boundaries, capturing global information with minimal computational overhead. The additional query should also facilitate efficient post-processing
Our solution, depicted in Figure involves a parallel boundary-wise segment proposal generation to yield compact and continuous segment proposals. By classifying these segment proposals via multiple query voting, our method, named BaFormer (Boundary-awal aware Transformer), realizes efficient TAS with high performance. Experiments conducted on three popular TAS datasets validate the effectiveness of our method. Specifically, on the 50Salads dataset: compared with the baseline ASFormer, our approach improves accuracy by 3.9%, Fl @50 by 7.9%, and edit score by 4.6%. Compared to the state-of-the-art performer DiffAct (2), our approach delivers competitive results while requiring only 10% of the FLOPPs and running time. Moreover, our query-based voting mechanism significantly reduces inference time to 24% of that required by the single-stage model UVAST 3]. The contributions of our work are threefold:
1. A query-based Transformer is introduced to convert TAS from dense per-frame classification into sparse per-query alternative, substantially reducing computational overhead.
2. The simple yet effective boundary-aware query voting ensures accurate and coherent action segmentation.
3. Experimental evidence demonstrates our model's superiority in efficiency over existing state-of-the-art TAS methods, without sacrificing effectiveness.
2 Related Work
Temporal Action Segmentation. In temporal action segmentation, many methods 1..
30
21 45| 6, 44
32 3 have been developed to address the challenge of over-segmentation. Notably,

---

MSTCN [5]]. ASFormer 45 and DiffAct [32 utilize repeating modules and multi-loss supervision to enhance long range dependency capture and refine preliminary predictions. In addition, several 18 incorporate post-proeessing techniques that integrate priors, such as boundaries 4 and transcripts 31, to further refine per-frame predictions. Additionally, other methods
methods 7
41
21
21 5 opt for parallel approach to frame-wise prediction and constraint learning within end end-to-end models. For instance, BCN [43 and ASRF [2I] introduce boundary prediction as auxiliary branch to enhance results, while UARL 66] leverages certainty learning to exploit transitional expressions. Despite these innovative efforts improving the performance, the reliance on multistage networks the backbone structure leads to increased computation costs. Building on recent advancements, UVAST 3 introduces a query-based network [40] to predict transcripts, transitioning from multi-stage to single-stage network for frame-wiss prediction, which results in a reduction of FL.OPs compared to earlier methods. However, post-processing with the Viterbi incurs significant time during inference, detracting from its suitability for real-time
43
algorithm 26 36
applications.
Query-based model for Image Detection and Segmentation. Benefiting from Transformer archi- tectures, query-based models have recently revolutionized object detection 4 34 13
33
46
which utilizes Transformer network to predict bounding boxes and their corresponding classes. This architecture can be readily adapted for image segmentation as well. MaskFormer [10] and Mask2Former [9] offer a direct application, that predicts mask classifications and their corresponding binary masks, transforming the task from pixel-wise dense prediction to mask classification. In recent advancements, SAM (Segment Anything Model) [24] emerges powerful solution for segmenting generic objects. Integrating query- bbaeed architectures has paved the way for more effective and versatile segmentation solutions.
Our method is different from previous TAS ones with the use of the query- model. UVAST [3]. yields a query-based transcript used for the potentially heavy post- pos--peeessing mostly related to ours. By contrast, our approach BaFormer utilizes the query-based module in conjunction with single- stage frame-wise module to decouple frame-wise results into query masks and classifications like that in image segmentation via mask classification, which is the first for TAS. The decoupling allows our method to facilitate efficient inference via query-baser majority voting, leveraging boundary information.
BaFormer
The overview of BaFormer is shown in Fig. 3 consisting of frame-wise encoder-decoder Transformer decoder, output heads, and inference processing. The input is a video & RTxx3XXxl comprising T frames of resolution H W, and the final output is frame-wise segment results.
In the frame-wise encodrr-decoder module, the frame encoder first extracts video feature maps F RT'xCo from the input video V. which are then fed to frame decoder with L layers followed by linear layer, resulting in Fd E RTx Additionally, we collect output features from L. frame decoder layers as (fi E RTTCE 1
Next, the Transformer decoder, comprising L layers, processes M trainable instance queries Qo E IRMM MMC, It takes features and masks M = (P e MT 1== as inputs. P. = r (Q,, F corresponds to the mask prediction probability estimated by the mask prediction head ipth Q is instance query embedding of the 1th Transformer layer. As shown in Fig. a), the 1th layer utilizes Q 1 f; to generate Q
In the output heads, each instance query embedding Q, is processed alongside frame embeddings F by three distinct heads, which calculate the probabilities for query class (Pf), masks (P)"), and boundaries (P)), respectively. Fig. 3 specifically illustrates the processing of the final layer's 's embeddings by the output heads.
Finally, during inference, the predictions are aggregate from the final Transformer layer, including query class probability Pī masks prediction p7, and boundary predictions P.. And aboundary-awe aware query voting employed to finalize the results.
3

---

Transformrr Decoder
Instance Quenes Ua
Q
Segmentation
Transformer
Classification Head go"
RE
Layer
MLP
M
Mask Prediction
Head q
P
Global query
Segment results
Prame Encoder
Frame Decoder
BD
Frame Decoder Layer xl.
pd
Boundary Prediction
Head o
P
Boundaries
Prume-wise Encoder-Decoder
33 Ourgut Heads
Inference
Figure 3: Overview of BaFormer architecture. It predicts query classes and masks, along with boundaries from output heads. Although each layer the Transformer decoder holds three heads, we illustrate the three heads in the last layer for simplicity.
3.1 Frame-wise Encoder-Decoder
The frame-wise encoder-decoder module is designed to preserve dense information essential for
model's functionality. Following previous research [ 45 32), we employ the I3D (5) encoder with frozen parameters to transfer video input V E R x into video features F RCx
setting Co to 2048. Subsequently, we select the ASFormer encoder [55. equipped with layers, as the frame-wise decoder. This decoder processes F to yield frame-wise embeddings Fd E R'''" enriching the model with dense information crucial for query masks and boundary predictions.
3.2 Transformer Decoder
The Transformer decoder aims to compress video sequences into sparse representations via queries to improve model efficiency. It stacks L Transformer layers {St f as illustrated in Fig. a). The output of the 1th layer Q1 E MC can be generated according to:
Q = PT f)), = 1,2,....
(1)
where f/ E TTC represents the frame-wise features from the 1hh frame decoder layer, and P =
p Q 1 F RXXT is the output of mask prediction head 4tth with input Q1-1 and Fd. The detailed computation of Th will specified in the output heads section.
The final output of the Transformer decoder, Q₂ is then directed to the output heads for the ultimate prediction. Additionally, the intermediate outputs Q: (for L) are also fed into output heads, serving for auxiliary outputs, which are further discussed in output heads and loss sections.
The specific structure of each Transformer layer Ω is illustrated in Fig. 4b). Each layer consists of three sub-layers with normalization and residual connection: (1) a Mask-Attenti Self-Attention (SA) layer, and (3) Feed-Forward Fetwork (FFN). The SA and FFN are the same as those in the vanilla Transformer [40]. The MA layer is a variant from Mask2Former [9]. where the thresholding within a temporally continuous feature space. Specifically, in the 1th Transformer layer, mask-attention layer in Fig. bb) can be formulated as:
(MA) layer, (2)
mask prediction is changed from hard into soft, which suits action segmentation
Linear Q:-1);
K Linear(fi),
V Linear(fi),
X, Linear softmax
Q,
Q
where @ is the Hadamard product. The mask pmm is probability matrix, with each element within the interval [o, 1). This structure enables the queries to concentrate more effectively on the principal
information of the frame sequence.
3.3 Output Heads
The output heads are designed to generate key elements required by the inference module, including query classes, query masks, and class-agnostic boundaries. Given (Q. there will be L groups of output heads. Each Q; is processed through the (h group of output heads to yield prediction. For

---

0c Add & Namm
Figure 4: Details of Transformer de- coder. (a) Transformer decoder stacks Transformer layers. (b) Each Trans- former layer consists of masked attention, self-attention, and a feed- forward network with residual connec- tions and normalization.
Tonsforncc
Layer 04
Tausforer
Layer Q
Tandfommt
Layer dg
Add & Nomi S2l otlention
0
X,
Misked
Pin
7-11
Q2-1
(a) Transformer decoder
(b) Transformer layer
notation conciseness, we omit the subscripts of the inputs Q₁ the output heads, and the predictions in this section.
Each group of output heads comprises three learnable modules: a classification head &-, mask prediction head " and a boundary prediction head sp" They will produce query class probability P E x query masks pm E IR MxT, and boundary probability pa E IR"" respectively. They can be formulated as:
P --= softmax (Q)), P== =' ( (LPP))
The Multi-Layer Perceptron (MLP) consists of three hidden layers. According to the M queries, denote output as D {pf M pm {p The predicted class-mask pair (p. )!")
corresponds to output of the jth query.
Before the instance query embeddings Q are fed into the boundary head, they are aggregated across the query dimension into global query by another MLP with three layers, followed by a linear layer, resulting in Q' Lineer(MMP((O) video information, thereby facilitating global boundary prediction. Then the boundary head can be formulated as:
RC, The global query encompasses comprehensive
pd p== 4 (Q', Fd) sigmoid Q TT
Moreover, outputs of previous Transformer decoder layers Q₂ < L.) are also fed into outputs heads through Equations (3) and to obtain auxiliary outputs Pf, Pf", and P/, serving as intermediate supervision.
3.4 Matching Strategies and Loss Function
Training the model requires ground truth supervision for query classes, masks, and global boundaries. For boundary detection, we represent ground-truth boundaries as yd E R²,, heatmap generated by applying Gaussian distribution to the binary boundary mask. For query class and mask ground truths, define ground-truth segments as class-mask pairs { ".") where yf E IK indicates the action class, y E IR is the corresponding binary mask, and N represents the total number of such segments in the video.
The predicted outputs for classes and masks, denoted as pe and pm. are organized into class-mask pairs (p.. p)))) 1 pf E k++ and p," E R²" represent the tt query's s class and mask probability prediction, respectively, and K is the number of action class in the dataset.
Before loss computation, crucial step match ground-truth data with query predictions. Depend- ing on the ground-trum segment type, we apply different matching strategies, as shown in Fig. 5
(1) Ordered Class Matching: instance queries are sequentially aligned with the class order, ensuring direct match. (2) Transcript Matching: instance queries are sequentially aligned with transcript ground truths by action order, with any excess query predictions aligned to no-action label. (3) Instance Matching: instance queries are matched with instance ground truths by Hungarian matching algorithm 277, with extra query predictions assigned no-action label. The first two strategies employ fixed matching based on query order, whereas the third strategy utilizes dynamic matching, requiring the matching cost from query class matching and mask matching. We define the cost when matching a pair of mask a E R² and b E R² as:
Lmack b) Atoect (a. b) + Adice (a,
(a. b),
(5)
where the Aiecal and Auice are the loss weight of focal loss [31 and dice loss (35 Defined as P( the probability value in the tt query class prediction pf is indexed by the 7th class. So the cost
5

---

Poskcisa of quary metitted with
Trikkeey
Poskeisa of sway raatched GT at closs
Cileganic
Allannaaaaono (a) Ordered Class Match
(b) Transcript Match
(c) Instance Match
Figure 5: Different matching strategies. Given example video including ordered action [ag, as. d1] from a dataset with all action classes {a+)==1> (a) and (b) are fixed matching, while (c) is dynamic matching.
between the th query prediction (p P and the j"h ground truth (2/) yj in Hungarian matching is:
Cost:j log (i.f Lamusk ,y"",
Experimental results reveal that instance matching outperforms others. Training losses for ordered class and transcript matching detailed in the supplemental material. When utilizing instance matching. the training loss for a video defined as follows:
M
N
T Σ
Σ
log P(aNC
Σ j=1
T Cmakk PbG))
2e p.
where (i) is the optimal ground truth index aligned with the ht query prediction, 8()) denotes the optimal prediction index corresponding to the jth ground truth, y is the tt element of yh, and pद is the 1th element pb.
3.5 Inference
During inference, a set of M pairs of query class-mask probability {Pf, P. )) are obtained, along with a boundary probability vector Pd IT,', from the th output heads. We apply dedicated algorithm outlined in Algorithm[[io derive the ultimate segmentation outcomes. Initially, boundary prediction produces class- cans-agnostic segment proposals. In each segment, each query mask within the segment is summed up and we identify the majority-contribue the class of this predominant query is assigned as the class for the segment. The algorithm is based on the observation that for each segment proposal span, all the queries may be activated in the mask prediction but the query corresponding the correct category always dominates.
query for the segment. Ultimately,
Algorithm I: Boundary-aware Query Voting
Input: Probability of query class-mask pairs: [pp...")) where pf e CR*1 p!" e R"; Boundary probability: p [p where
IR the boundary probability the th frame.
Output: Frame-wise segmentation: S IT.
Initialize S RD with all
{els, clls. = argmax (p K))),, [bb) %% sont (T U[d of 1)0 An))Itte) 1, ..., No do
for
1, 1.,...., do
Way Σ p. [b. beti]
end
A arggaax, , Sib. be+1] clsa
10 end
Experiments
4.1 Setup
Dataset and Evaluation Metric. We use three challenging datasets, GTEA 16), 50Salads [39 and Breakfast (25), where 50Salads presents the longest videos while GTEA includes the shortest

---

Match
FLOP
Time
Para
F1
Editt
Acc.
Query
[FLOP (C9
<0Ea,, 25,
F1
Ordered Class]
Transript
0.136
$7.0
83.5
48.2
87.9
AShormer
6.00
0.350
83.4
76.0
75.6
Instance"
4.23
0.144
85.3
84.6
79.9
79.8
$6.1
Class Token
4.54
0.139
1.63
55.7
84.5
75.9
79.7
86.3
Instance
100
4,45
0.139
1.63
89.3
$8.4
83.9
84.3
$9.5
Awsrgge Pasling
4:54
0.139
186
59.3
88.4
539
84.2
80.5
[+0.7]
40.22
+00013
-0.055
+0.14
20.14
+33.0
+1.2
453.3
+1.4
+35.7
+0.4
+29.7
+1.5
+29.7
+1.
Cla Sriee
-3.6
+3.0
+50
44.5
Table 2: Performance of different global queries on 50Salads.
Table 1: Comparative analysis of matching strategies
on 50Salads. (#Q: number of queries.)
ones, while Breakfast encompasses 1712 videos, which is the largest dataset. We conduct 4-fold cross-validation for GTEA and Breakfast and 5-fold cross-validation for 50Salads, consistent with 42 We assess accuracy for all frames as our key evaluation metric and report the segmental Edit score, and F1 scores for various Intersection over Union (loU) thresholds (10% 25% 50%) denoted as F1 @ ( [10, 25, 50).
previous research 21 30 4
45
15
Implementation Details. For the frame encoder, to obtain the frame-wise features with a dimension of 2048, The frame decoder allows any architecture designed for dense prediction tasks. In our paper, we utilize the ASFormer encoder 45], replicating its configuration with 11 layers and an output dimension C of 64. For the Transformer decoder, the initialization starts with 90 queries for GTEA and 100 queries for 50Salads and Breakfast datasets, subsequently employing 10-layer Transformer decoders. Each decoder layer comprises three attention heads and hidden dimension of 128. In the output heads, for mask and boundary prediction, employ MI.P layers with a hidden dimension of 64. As for Hungarian matching, Afocal = 5.0 and
use the pre-trained I3D S with fixed parameters,
Adice 1.0. During training, adopt Adam optimizer 23 with the step learning rate schedule
in 119. Initial learning rates are set to 5 x 10-4 for GTEA and 50salads datasets, while 1 x 10 for Breakfast dataset, incorporating a decay factor of 0.5. The training with 300 epochs utilizing batch size of 1. All experiments are conducted on an NVIDIA RTX 3090.
4.2 Ablation Studies and Analysis
We use boldface and underline for the best and second best performing methods in tables and indicate the performance improvements with A. All ablation studies are conducted on the 50Salads dataset. More analysis is provided in the supplemental material.
Analysis of matching strategies. We evaluate the impacts of different matching strategies sum- marized in Table We explored three matching approaches: Ordered Class Matching, Transcript Matching, Instance Matching. Instance utilizes the minimum feasible query count (26) align computational costs with the other methods. In the ordered class matching model, deterministic query-to-class assignments provide transparent optimization trajectory during training. Its com- mendable performance reflects its effectiveness. Instance', which employs bipartite matching to correlate queries with unique segment instances, displays slight decrease in performance relative to ordered class matching, with accuracy, Fl @ 10, and Edit distance decreased by 1.8%, 2.8%, and 2.9%, respectively. This modest divergence implies that fixed query-to-clan mappings may benefit training. However, unlike static class numbers, increasing the number of queries from 26 to 100 enhances the performance of the instance- based method. In contrast, the transcript matching strategy underperforms, evidenced by its considerable performance metric deficits. This technique, which encodes only the sequential information into each query, predisposes the queries towards positional grouping rather than convergent model learning, indicating that this is insufficient for coalescing actions into meaningful query clusters.
Effectiveness analysis of different kinds of global query. The global query generates boundaries that offer class-agnostic segment proposals essential for the query-based voting mechanism. We explore two approaches for deriving the global query: a class token and an average pooling token. The class token, like class token ViT [14, is learned at the beginning of the Transformer decoder, whereas the average pooling token is aggregated from instance queries by an average pooling operation. The comparative results of these approaches are presented in Table 2 When evaluating different strategies, it demonstrates that the class token is inferior to the average pooling token. We speculate that the superior performance of the average pooling token can be attributed its benefit from intermediate, query-based supervision, unlike the class token, which relies exclusively on
7

---

Voting strategy
Time(s)
F1
F
250,
Edit
Acc.
(F)
0.187
85.3
84.8
77.2
78.6
87.7
Query-based((V)
0.139
89.3
88.4
83,9
84.2
89.5
AQV-PV
-0.043
+4.0 +3.6
+6.7
+5.6
+1.8
Table 3: Performance and efficiency of different voting strategies on 50Salads.
Figure 6:
Query predictions and
frame-wise wise results on 50Salads.
Method
Time(s)
F1 (10,25,00
Edit
Acc.
Boundary
F
Edit
Acc.
NMS
0.138
89.1
88.4
84.0
83.8
89.1
Predict
89.3
88.4
83.9
84.2
89.5
peak
0.139
89.3
88.4
83,9
84.2
89.5
GT
91.8
91.8
90.2
88.3
95.9
peak NMS
+0.001
+0.2
0
-0.1
+0.4
+0.4
ArT-Predei
ArT Prediet
42.5
+3.4
+6.3 +4.1
+6.4
Table 4: Different strategies on boundary generation Table 5: Performance with predicted or on 50Salads.
ground-truth boundaries on 50Salads.
boundary supervision. This intermediate level of supervision likely results finer temporal resolution and more integrated global context, significantly improving the model's predictive accuracy.
Why do we use query-based voting? BaFormer efficiently compresses videos into sparse represen- tations, thereby reducing computational demands. To derive the final frame-wit wise results from the predictions of query classes, masks, and boundaries, we employ two voting strategies: frame-based and query-based voting. These strategies correspond to voting on boundaries by queries, and frames, respectively (details are provided in supplemental material). The results of these voting strategies are presented in T Specifically, the query-based voting mechanism identifies the query contributing most significantly to a segment, treating each query as an integral unit. Conversely, the frame-based voting method, detailed in the supplemental material, determines the segment's class label based on the majority class across frames within the segment. According to the comparison detailed in Table 3 query-basee voting significantly surpasses frame-based voting, showcasing improvements in running time and evaluation metrics. Specifically, accuracy obtains an enhancement of 1.8%. The edit score is increased by 5.6%, and F1 @50 is improved by 6.7%.
Further, we identify the reasons why query-based voting enhances the results and why it is superior to frame-based voting in Figure Figure 6 comprises two parts. Specifically, to generate the top part, we binarize the predicted masks from the final transformer decoder layer P E RMxT to or with threshold 0.5. Thus, the positions with value of I indicate the presence of actions corresponding to the query class. Next, we apply different colors to these positions and each color represents a single action. Then query masks are stacked to form the top part. In the bottom part, we present video segment results brame-bbsed and query-based voting, along with the ground truth. The red arrow points towards specific query within segment proposal, illustrated between two vertical black dashed lines. The red dashed box shows the segment results within the proposal. Query-based voting can identify new action segments, in contrast to frame-based voting, which is limited to recognizing only the predefined action classes within a segment. As demonstrated in Figure 6 we highlight intervals where actions discerned that would not be apparent through smoothing of frame-wise results. We hypothesize that this is due to the voting process among segmental candidates, which is independent of class probability and focuses instead the number of frames. This approach can rectify misclassifications inherent in frame-wise analysis that overemphasizes class probability.
Different strategies for boundary generation. To obtain boundaries quickly, experiment with two techniques: Non- Mani-mm Suppression (NMS) and peak choice, with the corresponding results presented in Table 4 Peak choice focuses on identifying boundaries within a local context. In contrast, NMS initially identifies boundaries with the maximum probability on global scale and subsequently eliminates other boundary candidates at local level. The analysis reveals that they yield similar outcomes and running time.
How well would our approach perform if we had perfect boundaries? We also conduct an analysis related to class-agnostic boundaries. Given that boundary quality is crucial to our process, we aim to quantify the potential performance improvement if "perfect" boundaries are available. To this end,
8

---

Method
Time
(4)
FLOPS
(G)
Param
(M)
GTEA
Ednt
Apc.
SUSaleds
EaRt
Azc.
FICTUIC55500
Breakfes
Fait
MSTIIN
[15
2010
0.994
4.99
0.993
55.8
83.4
69.8
79.0
76.3
16.5
74.0
64.5
62.9
80.7
52.6
48.1
17.9
61.7
66
SSTDAA
2020
0.173
9.37
0.80
00.0
89.
75.0
86.2
79.3
63.0
61.5
73.8
75.8
83.2
15.0
69.
55.2
73.7
70.2
BCN
2020
0.152
73.54
12.77
58.5
87.1
77.3
84.4
79.3
62.3
61.3
74.0
74.3
84.4
68.7
65.5
55.0
66.2
70.4
HASR
11
3021
0.217
20.02
19.17
50.9
38.6
76.4
87.5
78.7
86.6
65.7
78.5
$1.0
83.9
74.7
69.5
57.0
71.9
69.4
DYORM
41
2021
0.261
3.75
0.73
57.3
36.0
72.9
93.00
77.6
79.
75.9
06:
12.0
930
68.7
61.9
46.6
63.9
68.3
Mallph
ASRE
211
2021
0.161
7.45
1.30
19.4
87.8
79.88
985.7
97.3
84
83.5
72.3
19.3
64.5
74.3
68.9
56.1
72.4
676
18
2021
59.9
87.
75.8
994.6
78.5
80.5
78809
69.8
734
82.2
749
60.0
53.2
73.3
70.7
ASPrrma
431
2021
0.350
6.66
1.13
90.1
88.8
70.2
84.6
79.7
85.
K3.4
76.0
79.6
85.6
7600
70.6
ST.
75.0
73.5
LARL
2022
02.7
21.5
82.8
85.1
79.6
85.3
R3.5
17.8
78.2
84.1
65.2
50.4
47.4
66.2
678
DTT.
2022
0.403
6.66
1.13
87.1
65.7
78.5
50.5
86.9
78.8
74.5
62.9
77.7
75.8
KCK
2021
91.3
90.6
81.4
81.9
50.3
87.6
56.
19.3
51.4
76.9
72.6
063
26.1
LaConiert
2021
0-302
8:91
0.86
89.4
81.7
73.6
66-1
77.0
2,905
43.94
1.21
91.3
84.77
89.6
90.1
83.9
75.9
78.4
KARI
77.4
85.5
75.7
77.8
UVAST
3022
0.577
3,86
1.27
54.2
92.5
62.2
86.2
70.4
79.5
70.0
77.2
Stugle
UVAST
2022
4800888
1.10
91.3
81.0
81.7
81.4
71.5
77.1
69.
Table 6: Performance on GTEA, 50Salads, and Breakfast datasets. In terms of running time, BaFormer outperforms all methods except MSTCN, As for accuracy, BaFormer achieves comparable or better results. UVASTİ, UVAST, and UVAST* represent UVAST with alignment decoder, Viterbi, and FIFA. All FLOPs and running time are evaluated on 50Salads using the official codes in a consistent environment. We omit the running time and FLOPs on GTEA and Breakfast for simplicity as they are proportional to video length.
Method
Base
FLOPs(G)
Time(s)
Parameters(M)
F1@ (10, 25, 50)
Edit
Acc.
SSTCN
1.71
0.035
0.30
27.0
25.3
21.5
20.5
78.2
MSTCN
CNN
4.59
0.994
0.80
76.3
74.0
64.5
67.9
80.7
BaFormer
4.02
0.074
1.55
84.3
83.1
75.8
78.4
84,5
Aguemmer SSTCN
+2.31
+0.039
+1.25
+57.3
+57.8
+54.3
+57.9
6.3
ASFormer(Encoder]
2.23
0.110
0.38
53.1
51.4
47.0
43.3
85.7
DitfAct(1 step Decoder)
Transfrmer
7.78
1.647
1.21
48.3
45.3
35.6
36.5
74.2
UVAST(Aligament) BaFormer
3,86
0.577
1.27
86.2
81.2
70.4
83.9
79.5
4.54
0.139
1.63
89.3
88.4
83.9
84.2
89,5
+2.31
+0.029
+1.25
+36.2
+37.0
+36.9
+40,9
3.8
Table 7: Performance of methods with similar running time, employing the CNN or Transformer based frame decoder on the 50Salads dataset. To achieve comparable running time, DiffAct (I step) is adapted with an encoder and single-step decoder, and ASFormer with an encoder only is included.
run BaFormer using ground truth boundaries the 50Salads dataset instead of our predicted class-agnostic boundaries, as shown in Ta All the metrics have improved significantly. This indicates that our approach yields more promising results by higher-quality class-agnostic boundaries, which can be one of the future directions.
4.3 Comparisons with the State of the Arts
In Table 6 we categorize methods by multi-stage and single-stage. Our main comparison with the single-stage methods since our BaFormer adopts the same single-stage frame-wise decoder. Compared with UVAST with different alignment modes, BaFormer demonstrates superior efficiency, achieving the shortest processing time while outperforming it in most metrics. Notably. BaFormer requires only 24% of the time compared to UVAST with an alignment decoder (UVAST'), while enhances accuracy by 20.8%, 10%, and 8.4% on the GTEA, 50Salads, and Breakfast datasets, respectively. These results underscore the efficacy of BaFormer in decoupling frame-wise prediction into query masks and class prediction. Overall, BaFormer surpasses all multi-stage methods in terms of accuracy and achieves comparable results in Fl and Edit scores on all datasets. As for efficiency, compared to state-of- the-rr multi-stage methods, i.e., DiffAct, our method operates with only 6% of inference time and 10% of FLOPs, demonstrating a highly efficient approach to TAS.
Comparison with the methods with similar Running time. We compare our method with several others that exhibit similar running time in Table focusing on both CNN and Transformer-based frame-wis wise decoder. Specifically, 'CNN based frame-wise deocder" refers to the single-stage MSTCN structure, and Trranforrmrr based frame-wise decoder' refers to the single stage of ASFormer struc- ture. Within the CNN based frame-wist decoder, i..., SSTCN (15), BaFormer achieves outstanding

---

enhancements, surpassing MSTCN by 3.8% in accuracy, 10.5% in Edit score, and 8.0% in Fl @50 Notably, BaFormer significandly augments SSTCN's performance, yielding 54.3% increase in Fl @50. This comparison underscores BaFormer's single-stage capability to achieve high perfor- mance efficiently, maintaining the same inference time. When leveraging Transformer as the frame-wise decoder, we modify the number of stages in multi-stage methods to match the inference time of Baformer. This adjustment leads to a new version of ASFormer with a single encoder and DiffAct with simplified step decoder. Among these methods, BaFormer emerges as the top performer. This signifies that achieving frame-wise results through the decoupling of query masks and class prediction is more effective than direct frame-win wise prediction.
5 Limitations and Conclusion
BaFormer takes the query-based Transformer framework, which is known for its slow training convergence and data thirsty. We observe that training query-based Transformer converges slower than training frame-wiss approaches. For the discontinuous binary mask predictions, it may also be attributed to the limited data of action segmentation benchmarks. We hope that these factors can inspire future work.
In conclusion, we introduce BaFormer, novel boundary-aware, query-based approach for efficient temporal action segmentation. Contrasting most of the previous methods that rely on multi-stage or multi-step processing, BaFormer employs a one-step strategy. It simultaneously predicts the query-wise class and mask, while yielding global boundary prediction for segment proposals. We apply query-based voting for segment proposal classification. BaFormer offers a unique perspective for addressing TAS challenges by integrating grouping and classification techniques, representing a novel perception in the temporal action segmentation paradigm, emphasizing both efficiency and accuracy.
Acknowledgments
We thank all reviewers for valuable comments and suggestions. The work was supported in part by US National Science Foundation Grants 2006665, 2128350, and 2331769. This work is also supported in part by Air Force Research Laboratory FA 9550-23-2-0002. The support of these agencies is gratefully acknowledged. Any opinions, findings, and conclusions, or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation or the United States Air Force.
References
[1] H. Ahn and D. Lee. Refining action segmentation with hierarchical video representations. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 16302- 16310, 2021.
[2] E. Bahrami, G. Francesca, and Gall. How much temporal long-term context is needed for action segmentation? In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10351-10361, 2023.
[3] N. Behrmann, S. A. Golestanch, Z. Kolter, Gall, and M. Noroozi. Unified fully and timestamp supervised temporal action segmentation via sequence sequence translation. In ECCV, 2022.
[4] N. Carion, Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko. End-to-end object detection with transformers. In Computer Glasgow, UK, August 23-28, 2020, Proceedings, Part I 16, pages 213-229. Springer, 2020.
2020: 16th European Conference,
[5] J. Carreira and A. Zisserman. Quo vadis, action recognition? new model and the kinetics dataset. In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6299-6308. 2017.
[6] I.. Chen, M. Li, Duan, Zhou, and J. Lu. Uncertainty-aware representation learning for action segmentation. In IJCAI, volume page 6, 2022.
10

---

[7] M.-H. Chen, B. Li, Y. Bao, G. AlRegib, and Z. Kira. Action segmentation with joint self- supervised temporal domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9454-9463. 2020.
[8] Q. Chen, X. Chen, J. Wang, S. Zhang, K. Yao, H. Feng, J. Han, E. Ding, G. Zeng, and J. Wang. Group detr: Fast detr training with group-wise one-to-many assignment. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6633-6644 2023.
[9] B. Cheng, A. Choudhuri, I. Misra, A. Kirillov, R. Girdhar, and A. G. Schwing, Mask2former for video instance segmentation. arXiv preprint arXiv:2112.10764 2021.
[10] B. Cheng, A. Schwing, and A. Kirillov. Per-pixel classification not all you need for semantic segmentation. Advances in Neural Information Processing Systems, 34:17864-1788 2021.
[11] R. T. Collins, A. Lipton, and T. Kanade. Introduction to the special section on video surveillance. JEEE Transactions pattern analysis and machine intelligence, 22(8):745-746, 2000.
[12] R. Т. Collins, A. J. Lipton, T. Kanade, H. Fujiyoshi, D. Duggins, Y. Tsin, D. Tolliver, N. Enomoto, O. Hasegawa, P. Burt, et al. system for video surveillance and monitoring. VSAM final report, 2000(1-68):1, 2000.
[13] X. Dai, Y. Chen, J. Yang, P. Zhang, L. Yuan, and L. Zhang. Dynamic detr: End-to-end object detection with dynamic attention. In Proceedings of the IEEE/CVE international conference computer vision, pages 2988-2997, 2021.
[14] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, Zhai, Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.
2020.
[15] Y. A. Farha and Gall. Ms-ten: Multi-stage temporal convolutional network for action segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 3575-3584, 2019.
[16] A. Fathi, Ren, and M. Rehg. Learning to recognize objects in egocentric activities. In CVPR 2011, pages
IEEE, 2011.
[17] S.-H. Gao, Q. Han, Z.-Y. Li, P. Peng, L. Wang, and M.-M. Cheng. Global2local: Efficient structure search for video action segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16805-16814, 2021.
[18] D. Gong, J. Lee, D. Jung, S. Kwak, and M. Cho. Activity grammars for temporal action segmentation. Advances in Neural Information Processing Systems. 36, 2024.
[19] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770- 778, 2016.
[20] J. Hong. H. Zhang, M. Gharbi, M. Fisher, and K. Fatahalian. Spotting temporally precise, fine- grained events in video. In European Conference on Computer Vision, pages 33-51. Springer, 2022.
[21] Y. Ishikawa, S. Kasai, Y. Aoki, and H. Kataoka. Alleviating over-segmentation errors by detecting action boundaries. In Proceedings of the IEEE/CVE Winter Conference on Applications of Computer Vision, pages 2322-2331, 2021.
[22] B. Jiang, Y. Jin, Z. Tan, and Y. Mu. Video action segmentation via contextually refined temporal keypoints. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11836-13845, 2023.
[23] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6600 69800, 2014.
[24] A. Kirillov, F. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. Berg, W.-Y Lo, et al. Segment anything. arXiv preprint arXiv:2304.0266 2023.
11

---

[25] H. Kuehne, A. Arslan, and T. Serre. The language of actions: Recovering the syntax and semantics of goal goal-directed human activities. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 780-787, 2014.
[26] H. Kuehne, A. Richard, and J. Gall. A hybrid ran-hmm approach for weakly supervised temporal action segmentation. IEEE transactions on paltern analysis and machine intelligence, 42(():765--75) 2018.
[27] H. W. Kuhn. The hungarian method for the assignment problem. Naval research logistics quarterly, 2(1-2):83-97) 1955.
[28] C. Lea, M. D. Flynn, R. Vidal, A. Reiter, and G. D. Hager. Temporal convolutional networks for action segmentation and detection. In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 156-165, 2017.
[29] P. Lei and S. Todorovic. Temporal deformable residual networks for action segmentation in videos. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6742-6751,2011 2018.
[30] S.-J. Li, Y. AbuFarha, Y. Liu, M.-M. Cheng, and J. Gall. Ms-tcn++: Multi-stage temporal convolutional network for action segmentation. IEEE transactions on pattern analysis and machine intelligence, 2020.
[31] T.-Y. Lin, P. Goyal, R. Girshick, K. He, and P. Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pages 2980-2988, 2017.
[32] D. Liu, Q. Li, A.D. Dinh, T. Jiang M. Shah, and C. Xu. Diffusion action segmentation. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pages 10139- 10149, 2023.
[33] S. Liu, F. Li, H. Zhang, X. Yang, X. Qi, H. Su, J. Zhu, and L. Zhang. DAB-DETR: Dy- namic anchor boxes are better queries for DETR. International Conference on Learning Representations, 2022.
[34] S. Liu, T. Ren, J. Chen, Z. Zeng. H. Zhang, L. Huang, H. Su, Zhu, et al. Detection transformer with stable matching. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6491-6500 2023.
[35] F. Milletari, N. Navab, and S.-A. Ahmadi. V-net: Fully convolutional neural networks for volumetric medical image segmentation. In 2016 fourth international conference 3D vision (3DV), pages 565-571. leee, 2016.
[36] A. Richard, H. Kuchne, A. Iqbal, and J. Gall. Neuralnetwork-iieebb:: A framework for weakly supervised video learning. In Proceedings of the IEEE conference Computer Vision and Pattern Recognition, pages 7386-7395, 2018.
[37] B. Roh, J. Shin, W. Shin, and Kim. Sparse detr: Efficient end-to-end object detection with learnable sparsity. arXiv preprint arXiv:2111 ftti:://11.13300.. 2021.
[38] S. Sayed, R. Ghoddoosian, B. Trivedi, and V. Athitsos. A new dataset and approach for timestamp supervised action segmentation using human object interaction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3132-314 3441, 2023.
[39] S. Stein and S. J. McKenna. Combining embedded accelerometers with computer vision for recognizing food preparation activities. In Proceedings of the 2013 ACM international joint conference on Pervasive and ubiquitous computing, pages 729-738, 2013.
[40] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[41] D. Wang. D. Hu, X. Li, and D. Dou. Temporal relational modeling with self-supervision for action segmentation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 2729-2737, 2021.
12

---

[42] P. Wang and H. Ling. Dir-as: Decoupling individual identification and temporal reasoning for action segmentation. arXiv preprint orXiv:2304.021100 2023.
[43] Z. Wang, Z. Gao, L. Wang, Z. Li, and G. Wu. Boundary-aware cascade networks for temporal action segmentation. In European Conference Computer Vision, pages 34-51. Springer, 2020.
[44] Z. Xu, Y. Rawat, Y. Wong, M. S. Kankanhalli, and M. Shah. Don't pour cereal into coffee: Differentiable temporal logic for temporal action segmentation. Advances in Neural Information Processing Systems, 35:1 15:14890-14903, 2022.
[45] F. Yi, H. Wen, and T. Jiang. Asformer: Transformer for action segmentation. BMVC, 2021.
[46] Z. Zong, G. Song, and Y. Liu. Detrs with collaborative hybrid assignments training. In Proceedings of the IEEE/CVF international conference on computer vision, pages 6748-6758, 2023.
13

---

Appendix
A
Broader impact
We assert that our proposed method processes temporal action segmentation efficiently while strictly preserving privacy by utilizzng features rather than raw video. Our work not only opens new horizons for the academic community but also advances the task towards real-world application. We also acknowledge that ethical concerns may be caused by the unsatisfactory boundary detection. Nonetheless, safety and reliability will be our top priorities when we deploy this system in real-world applications.
B
Notations
To clarify, we denote as the index for the transformer decoder layer and as the index for the query. Subscripts indicate the indices of parameters, while superscripts serve abbreviations representing parameter types. For instance, the superscript m p" stands for 'mask.'
Notation
Meaning
F Video features extracted from frame encoder Outputs of frame-wise encoder-decoder module
Fd
F
Features collections from L frame decoder layers
f
Output features of thh frame decoder layer
Qo
M trainable instance queries with random initialization Instance query embedding from the 1h layer
Q
P
Query class prediction from the 1 layer (the represents 'class")
Pr
Query masks prediction from the 1d layer (the represents 'mask')
ph
Boundary predictions from the 1hh layer (the represents 'boundary') The th query class prediction in P (the index / is committed in P for simplicity)
p
p
The 2h query mask prediction in pm (the index is committed in p for simplicity) The boundary prediction in the 1h frame
p
oe
The head for predicting query classes The head for predicting query masks
The head for predicting boundaries
34
The ground-truth action class for the 8th query
The ground-truth binary mask for the hth query The ground-truth boundaries
y
C Other implementation details
C.1 Loss for ordered class and transcript matching
Similar to instance matching, we model ground-truth boundaries as yd E IR",, heatmap generated by applying a Gaussian distribution the binary boundary mask. For ground truths of query classes and masks, we employ different strategies: In ordered class matching, segments are derived according to categorical order. In contrast, transcript matching segments are determined by the action sequence outlined in the video transcript. Consequently, we represent ground-truth segments as class-mask pairs (vff, y 1 with of denoting the action class, y E R as the corresponding binary mask, and N as the total number of segments in the video. In ordered class matching, the N = K, where is the total number of action classes in the dataset. the transcript matching, the N the number of actions in the video transcript.
14

---

The predicted outputs for classes and masks, denoted as P and P",, are organized into mask pairs {Pp }, = where d RK+1 and p." R" represent the 7t query's predicted class probabilities (including no-label class) and mask probabilities, respectively. We define P((J) as the probability value in the 7th query class prediction p indexed by the fuh class. We define the cost when matching a pair of mask a E R and TR." as:
maskk (a, b) Srocal a. b) + Auice (a. b),
(8)
where the Arcaal and Adice is the loss weight of focal loss [31 and dice loss 335
As result, the loss function in ordered class and transcript matching can be formulated as follows:
C Σ (Cmask(pp .Y?)) log ((
Σ ye log lo
(9)
where M is the number of queries, T is the length of the video, v/ is the tth element of yf, and p is the th element of P", which is the prediction of boundary probability.
C.2 Boundary-aware frame-wise voting
Different from the query-based voting, we also explore frame-wise voting, a more direct approach when starting with frame-wise results. The details of the algorithm are in Algorithm 2
Algorithm 2:
Boundary aware Frame Voting
Input:
Probability of query class P M))+ Probability of query masks pm R.MT; Boundary probability: pd [pi]T= where pf e R is the boundary probability in the tt frame.
Output: Frame-wise segmentation: S E R...
Initialize S E RT with all zeros
p (Pm) (P F; KJ) E RT
So = argmax PP)) E IRT
B (b,), sort
> pl-1)& pl)))
for = 1,2, N₂ do
act id FindMaajrity (Solb: b(41)) Sbb b id
8 end
D Impact of parameters
D.1 Number of transformer decoder layers
T explores the impact of varying the number of transformer decoder layers on the 50Salads dataset and reveals a trend correlating the number of layers with performance metrics. Reducing the transformer decoder layers
results in a significant reduction of FLOPs to 2.97G and a minimal change in running time, but it leads to decreased performance across all metrics, indicating a trade-off between a more lightweight model and diminished robustness.
further increase to 10 layers results in the best performance metrics, surpassing the model with 8 layers by 1.1% on accuracy, indicating that the additional depth provides a beneficial complexity that enhances the model's learning capability. Overall, the results suggests that while increasing the number of transformer decoder layers incurs more computational cost, it also significantly boosts performance, indicating trade-off between efficiency and efficacy.
D.2 Query quantity
Table providess an insightful look into how the quantity of queries impacts both the computational cost and the performance of a model. Starting with 50 queries, see computational cost of
15

---

# Transformer' decoder layer
FLOPs
Running time(s)
#Parameter (M)
Edit
Acc.
(G)
@ (10, 25, 50)
2.97
0.133
0.78
82.8
81.6
76.5
77.2
84.2
3.42
0.136
1.02
86.4
85.6
80.7
79.8
86.3
8
4.09
0.137
1.38
88.6
87.1
83.1
83.4
88.4
10
4.45
0.139
1.63
89.3
88.4
83.9
84.2
89.5
Table 8: Results of different numbers of Transformer decoder layers on 50Salads.
FLOPs
Running time(s)
# Parameter (M)
F1
query
(G)
@
[10, 25, 50)
Edit
Acc.
50
4.26
0.138
1.629
87.8
87.2
82.8
82.8
87.1
70
4.28
0.139
1.630
88.4
87.5
83.4
82.9
87.9
100
4.45
0.139
1.632
89.3
88.4
83.9
84.2
89.5
120
4.56
0.139
1.633
86.6
85.7
81.1
81.3
87.0
150
4.60
0.139
1.636
84.3
82.8
77.2
78.4
84.4
Table 9: Influence of query quantity on 50Salads.
4.26G FLOPs and accuracy of 87.1% %. Increasing the query count 70 results in marginal increase in FLOPPs to 4.28G, but a notable improvement in accuracy to 87.9%. This indicates that slight increase in the number queries can enhance the model's ability without significantly raising computational costs. However, the model's performance peaks at 100 queries, achieving the highest accuracy of 89.5% along with the best Fl and Edit scores, a computational cost of 4.45G FLOPs. This suggests that there is an optimal query range where the model's ' performance maximized. Interestingly, beyond this point, as the number of queries continues rise 120 and 150, there is diminishing return in terms of accuracy, showing that additional queries beyond a certain threshold may instead lead to overfitting.
E
Structure and loss
E.1
Single/Multiple features
Different connection between frame decoder and transformer decoder. We investigate diverse strategies for integrating the frame decoder with the transformer decoder modules. In the simplest form, a uniform connectivity strategy is employed, wherein the transformer decoder ingests shared video features originating from a singular level of the frame decoder, as illustrated in Figure aa).. Conversely, a more sophisticated approach is explored, wherein the transformer decoder harnesses hierarchy of video features, sourced from multiple layers within the frame decoder, to enrich the representational capacity of the system, as demonstrated in Figure (b.
The results of these connectivity strategies are detailed in Table 10 We include the ASFormer as baseline for comparison. Regardless of whether auxiliary losses are applied not, the adoption of multi-level features yields substantial and noteworthy performance enhancements. These performance improvements come at a minimal cost terms of additional parameters and FLOPs. Specifically, we observe an increase in accuracy by 5.5% and FI @ 10 by 6.0%, along with corresponding gain accuracy by 3.9% and F1 10 by 3.9% in the presence and absence of auxiliary losses, respectively. This phenomenon can be attributed to the mutually beneficial relationship between query and mask predictions. When the transformer decoder and frame decoder operate independently, there is deficiency in the exchange of crucial information, impeding the training of query- query-based mask classification methods.
E.2 Auxiliary loss
To rigorously analyze the impact various loss functions on our method, we provide comparative results in Table I0 detailing performance metrics both with and without auxiliary losses. These
16

---

Transformer Decoder
Transformer Decoder
queries
qqueries
video
features
video
features
Frame Decoder
Frame Decoder
(a) Single-level features connection
(b) Multi-level features connection
Figure 7: (a) and (b) illustrate the single-level and multi-level feature connection strategies, respec- tively. In (a), a single-level feature from the frame decoder is shared with the transformer decoder layers. While (b) involves the integration of multi-level features from various layers of the frame decoder. (Note: Mask inputs have been omitted for simplicity.)
Auxiliary loss
@Parameter (M)
Running
FLOPPs (G)
F1
Feature
Edit
Acc.
Time(s)
10,
25,50)
ASFormer
1.13
0.359
6.66
85.1
83.4
76.0
79.6
85.6
single
no 1.51
0.137
3.83
82.6
81.8
75.6
76.6
83.5
BaFormer
yes
83.3
82.2
76.5
78.6
84.0
multiple no 1.63
0.139
4.45
86.5
85.3
81.2
80.4
87.4
yes
89.3
88.4
83.9
84.2
89.5
Table 10: Comparative analysis of the effect of feature connections, iee., single multiple, 50Salads and the use auxiliary loss.
results indicate that auxiliary losses enhance performance in both single-level and multi-level feature representations. Notably, applying auxiliary losses in multi-level features yields substantial improve- ment in accuracy (2.1%) 1)), surpassing the 0.5% accuracy gain observed in single-level features. This suggests that auxiliary losses have a more pronounced effect on multi-level feature representations.
More detailed efficiency comparison and experiments
F.1 Methods with various stages.
Table II presents a comparative analysis of various stulti-taggesstpp methods contrast to BaFormer for Temporal Action Segmentation (TAS). The methods compared include MSTCN with varying stages, ASFormer with different numbers decoders, and DiffAct with varying inference steps. We have calculated the FLOPs and parameter count utilizing the official code repositories. We employed the DiffuAct model, as is reported to achieve the final results in their publication, to determine the FLOPs and parameter metrics.
BaFormer has moderate FLOPs (4.45 G) and parameter count (1.64M), which stands between lightest and heaviest models compared. is considerably more efficient than the most computationally intensive models like DiffAct with 25 inference steps. MSTCN and ASFormer models show trend where increased stages decoders generally lead to better performance, but also increased computational costs. DiffAct's performance improves significantly with the number of inference steps, yet this comes at the cost of substantial increase in FLOPs.
In conclusion, BaFormer achieves balance between computational efficiency and performance, making it a compelling choice for applications where both are critical considerations and presents competitive alternative to existing multi-stage/step methods, offering a reduction computational demands while maintaining high accuracy and outperforming in critical metrics like Fl score and Edit distance. This suggests that BaFormer's approach to integrating hierarchical tokens and utilizing boundary-aware transformer architecture is effective for TAS tasks.
17

---

#Stage /Step
FLOPs (G)
Running time(s)
#Parameter (M)
F1
Method
Edit
Acc.
@ [0..
25, 50}
MSTCN
1.71
0.080
0.30
27.0
25.3
21.5
20.5
78.2 2
MSTCN²
2
2.67
0.086
0.47
55.5
52.9
47.3
47.9
79.8
MSTCN'
3
3.63
0.092
0.63
71.5
68.6
61.1
64.0
78.6
MSTCN
4.59
0.094
76.3
74.0
64.5
67.9
80.7
ASFormer
2.23
0.168
0.38
53.1
51.4
47.0
43.3
85.7
ASFormer
3.71
0.235
0.63
79.5
77.4
71.6
71.5
86.8
ASForme
3
5.19
0.299
0.88
83.9
82.8
76.8
76.7
86.8
ASFormer
4
6.66
0.359
1.13
85.1
83.4
76.0
79.6
85.6
DiffAct
7.78
1.647
1.21
64.9
63.8
59.3
56.5
88.6
DiffAct*
12.30
1.784
1.21
87.6
86.6
81.2
82.1
89.1
DiffAct
8
18.33
1.887
1.21
89.3
88.3
83.1
83.5
89.0
DiffAct
16
30.38
2.043
1.21
90.0
88.8
83.3
84.5
89.0
Di
25
43.94
2.306
1.21
90.1
89.2
83.7
85.0
88.9
BaFormer
4.45
0.139
1.63
89.3
88.4
83.9
84.2
89.5
Table 11: Comparative overview of multi- methods versus BaFormer 50Salads. Here, "MSTCN"" and "ASFormer" denotes model with n processing stages, while "DiffAct" signifies model with "n" decoder steps.
Segment length
0~~000
1001~~000
2001~3100
Frame-based
87.68 86.85
90.98 92.01
92.47
Query-based
95.38
Table 12: Accuracy for action segments of different lengths, comparing basne-baedd and query- -ased methods on the 50Salads dataset.
F.2 Performance on segments with different lengths
To further compare the results of the frame-based and query-based voting methods, we propose analyzing the outcomes at the segment level rather than across entire videos. We conduct experiments on action segments of varying lengths based on the ground truth annotations of 50Salads dataset, and the results are summarized below in Table 12 We observe that frame- based methods perform slightly better shorter segments, while query-based methods demonstrate higher accuracy on longer segments. We attribute this difference the voting mechanism. Specifically, when applying the same boundaries for both methods, the query- besedd approach relies on query-level classification and masks. This strategy captures information at the segment level, as opposed to frame-based methods, which depend on individual frame-level predictions, leading to distinction in performance.
G More visualization
In the 50Salads dataset, we present enhanced visualizations for a more comprehensive analysis. Each subfigure's 's upper portion depicts the outcomes of instance segmentation, while the lower portion compares the segmentation results without boundary integration ("F"), the results achieved by BaFormer ("S""), and the ground truth ("GT"). The absence of boundary consideration ("F"), where the predicted instance token class probability is directly combined with the mask probability, leads to excessive over- sermeeemmntttion and suboptimal boundary delineation. This effect is attributable the wise basis of mask prediction, which, when applied directly, perpetuates the over-segmentation issue. "F"' with "S" illustrates that BaFormer's advancements are realized through the reduction of over-segmentation and the refinement of boundaries. This improvement stems from the more continuous nature of segment proposals informed by boundary predictions, enhancing the assignment of actions to each proposal. The transition from the results of instance segmentation to frame-wise
frame-wis
18

---

(a) "rgh-03-1" 50Salads
(h) "rgh-07-2" 50Salads
(c) "rgh-22-22 50Salads
(d) 255--25-2" 50Salads
Figure 8: Visualization of the 50Salads dataset. Each subfigure presents comparison of instance segmentation and frame-wise results. "F" indicates the absence of boundary utilization. "S" signifies its inclusion. "gt" represents the ground truth.
outcomes elucidates the efficacy of the majority voting mechanism. (Note: Boundary prediction is not depicted in these visualizations.)
19

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: In the abstract and introduction, we discuss the challenges of the task and present the solutions offered by our methods. The contributions are highlighted in the introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The limitation discussed in the part of "Limitations and Conclusion".
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
20

---

Answer: [NA]
Justification: The conclusions presented in this paper are based on experimental results and do not include theoretical findings.
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer:[Yor Yes]
The model can be replicated using the implementation details provided in the paper and supplementary materials.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
21

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We utilized public datasets for our experiments, and the code will be released upon the paper's acceptance. Additionally, the details provided in the paper and supplemental material will aid in reproducing the experimental results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
mublic/guided
poodeeSubmissioooollee
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
//nips wwtp::ccpppp.ggiccee/c////////// Codeeubmissiionn iwrrbbmmssioooooolioce
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: The paper specifies all the datasets and the splits used for training and testing. Detailed information on the training process, including hyperparameters, optimizer settings, and other relevant details, is provided in both the paper and the supplemental material.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: In the ablation study section, the part titled "How well would our approach perform if we had perfect boundaries?" demonstrates the potential improvement in model performance if boundary errors were reduced.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
22

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CL, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The paper provides details on resources, model parameters, and running time.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: The paper uses the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
See the "Broader Impacts" in the supplementary.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
23

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We cite the original paper that produced the code and dataset, shown in the paper and reference.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
24

---

If assets are released, the license, copyright information, and terms of use in the thcode has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssit
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer:[NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The datasets used in the paper are from public sources.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The datasets used in the paper are from public sources.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
25

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
26


Reviews:
[{"review_id": "JdgzgIw3yN", "reviewer": "Reviewer_rrtF", "summary": "The paper introduces BaFormer, a boundary-aware Transformer network designed to enhance the efficiency of Temporal Action Segmentation while maintaining high performance. BaFormer tokenizes each video segment as an instance token for intrinsic instance segmentation and employs instance queries for segmentation and a global query for class-agnostic boundary prediction. This approach allows for continuous segment proposals and a simple voting strategy during inference.", "questions": "The questions have been detailed in the \"Weaknesses\".", "limitations": "None", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1, The proposed Query Voting method, though simple, is highly effective as demonstrated by the ablation experiments in Table 3.\n\n2, The proposed BaFormer being a single-stage method is a practical advantage.\n\n3, The proposed BaFormer achieves performance close to state-of-the-art two-stage methods.", "weaknesses": "1, One of the major innovations, Boundary-aware Query Voting, is implemented only during the inference stage, and its process is very simple. The method involves summing up query masks within segment proposals and selecting the highest, which is a common approach in many temporal action proposal works. This raises doubts about the innovation's originality. Additionally, BaFormer's overall network design follows previous works, with the main innovations being in the Matching Strategies and Voting.\n\n2, Besides I3D, the authors should compare the impact of other visual encoders on the model's performance. Understanding the effects of different visual features is crucial, and the visual encoder used in Table 6 should be clearly stated to ensure fair comparisons.\n\n3, In Table 7, the terms \"CNN and Transformer backbones\" need clarification. The results for BaFormer with a Transformer-Base backbone correspond to those in Table 6, suggesting that \"backbone\" might refer to the Frame Decoder. This is inconsistent with the \"Frame-wise Encoder-Decoder\" terminology used by the authors.\n\n4, The authors incorrectly cite KARI[18] as a NeurIPS 2024 paper, whereas it is actually from NeurIPS 2023. It seems the authors are not aware that they are submitting to NeurIPS 2024 themselves :)\n\n5, Figure 6 aims to explain why Query Voting is superior to frame-based voting, but the visualization of queries and the meaning of different colors are not clearly explained, causing confusion for readers. Additionally, it would be interesting to know the performance of frame-based methods on shorter segments, as they might perform better than query-based methods in such cases.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments. We provide discussions and explanations about your concerns as follows.\n\n**1. For Weakness1**\n\n(**Query voting is very simple**) We thank you for acknowledging our contributions to Matching Strategies and Voting. Regarding your concern about the simplicity of the process, we intentionally designed it to be simple to contribute to efficiency.  In our paper, we aim to design an effective TAS model with high efficiency. So when designing such a model, we pursue two key points: (1) a highly efficient model structure to avoid heavy multi-stage or multi-step architecture; (2) integration of simple but effective post-processing upon high-quality model predictions. Specifically, our model transforms frame-wise predictions into query class-mask predictions for TAS. At the same time, we put much effort into exploring different matching strategies and extensively ablate different components (number of decoder layers, query quantity, auxiliary loss, global query for boundary generation, etc., as reported in experiments and supplementary material.) of the network architecture. **After achieving high-quality query class-mask predictions as well as precise boundary generation, we are then enabled to use very simple but highly efficient query voting to realize the expected efficient TAS**.\n\nWe can also adopt more complicated post-processing, such as the Viterbi algorithm, activity grammar, and so on. However, previous papers have verified that they are time-consuming, which is not what we desired. For example, in another single-stage method UVAST, when using the Viterbi algorithm it spends more than **480s** with an accuracy of **87.4%** on 50Salads dataset; while when using simpler post-processing, i.e., alignment decoder, it takes **0.577s** but with a degraded accuracy of **79.5%**. By contrast, our BaFormer employs simple voting with **0.139s** and achieves a higher accuracy of **89.5%**. \n\n(**BaFormer follows previous work**) As for the concern that BaFormer follows previous work, BaFormer is designed to improve efficiency, and we experientially find that a query-based model can change the dense information into sparse information to reduce FLOPs. So, we follow the DETR-style pipeline to predict the query-based results. It gives us insight to take a new paradigm in TAS. **Our focus is on how to innovatively adopt the DETR-style model into TAS that has not been fully explored by such an approach**. Like BaFormer, many other promising works also follow the DETR model to achieve their different targets, for example, MaskFormer in image segmentation, CLTR  in crowd localization, TrackFormer  in multi-object tracking, and so on. They design new pipelines similar to DETR for different tasks as we do.\n\n **2. For Weakness 2**\n\n(**Other visual encoder**) Thank you for the suggestion to adopt other visual encoders, which will enrich our model's performance. Accordingly, we conducted experiments with TSM visual encoders compared with I3D. The results on 50Salads are as follows:\n\n|    Visual Encoder      |   F1@{10,20,50}  | Edit | Accuracy |\n|:--------:|:----------------:|:----:|:--------:|\n|    I3D | 89.3 | 88.4 | 83.9 | 84.2 | 89.5 |\n|    TSM | 90.7 | 89.7 |  85.6 | 86.0 | 90.8| \n\nThe results indicate that a superior visual encoder can further enhance performance.\n\n **3. For Weakness 3**\n\n(**Terminology**) Sorry for the confusion. In our study, the term \"CNN backbones\" refers to the single-stage MSTCN, and the \"Transformer backbone\" refers to the single stage of ASFormer. In \"Frame-wise Encoder-Decoder\", the structure of the frame-wise encoder is fixed as I3D for all methods in Table 7. Therefore we focus on different structures of the frame-wise decoder. As acknowledged by the reviewer, the term \"backbone\" specifically refers to the frame-wise decoder. We apologize for any confusion and thank the reviewer for pointing it out. For clarity, we have replaced the term \"backbones\" with \"frame-wise decoder\".\n\n **4. For Weakness 4**\n\n(**Typo in reference**) Thank you for pointing out this embarrassing typo. We promise to fix it and check carefully other references in the revision.\n\n **5. For Weakness 5**\n\n(**Visualization in Figure 6**) Again, sorry for the confusion, and we will add relevant clarification in the revision. Figure 6 comprises two parts. Specifically, to generate the top part, we binarize the predicted masks $\\in R^{n\\times T}$ to 0 or 1 with a threshold 0.5, where $n$ is the number of queries, and $T$ is the length of the video. Thus, the positions with a value of 1 indicate the presence of actions corresponding to the query class. Next, we apply different colors to these positions and each color represents a single action. Then query masks are stacked to form the top part. In the bottom part, we present video segment results of frame-based and query-based voting, along with the ground truth. The red arrow points towards a specific query within a segment proposal, illustrated between two vertical black dashed lines. The red dashed box shows the segment results within the proposal.\n\n\n(**Performance on segments with different lengths**) Thanks for the suggestion. We conducted experiments with different segment lengths and the results are summarized below. We find that the performance of frame-based methods is slightly better on shorter segments, and the accuracy is shown below:\n\n\n|  Length   |0~1000 | 1000~2000 | 2000~ 3100 |\n|:--------:|:----------------:|:----:|:--------:|\n|    Frame-based | 87.68 | 90.98 | 92.47|\n|    Query-based | 86.85 | 92.01 | 95.38|", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wdKwbIQKuC", "reviewer": "Reviewer_W81q", "summary": "This paper proposes a fully-supervised TAS approach that predicts both segment and frame predictions. Two levels of predictions are then combined to form the final action boundaries. The approach achieves competitive performance and at the same time reduces inference time for better effiency.", "questions": "1. What is the advantage of leveraging an extra branch of segment predict? What advantage did it bring in terms of accuracy and combating the issue of over-segementation? \n\n2. Clarifications reagarding the confusing points listed in weakness.", "limitations": "The authors had discussed in the supp.", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. The motivation of leveraging segment-wise prediction to help reduce the temproal dimension processed by model is sound. The design drawing inspiration form image classification looks soild for TAS.\n\n2.  The apporach successfully boosted the processing efficiency of action segmentation while maintaining high accuracy on common TAS benchmarks.", "weaknesses": "The reviewer understands that the authors' main objective claimed is to reduce the processing time. However, given the design of voting between frame-wise and segment-wise predictions, one expected advantage of having access to segment prediction is to reduce the over-segmentation issue, yet the effect of such voting scheme between frame-wise and segment-wise predictions did not achieve the expected effect. On most of the metrics, the propose approach did not manage to outperform existing SOTA approaches, for example DiffAct, especooally on the segmental metrics. \n\nThe paper is diffult to parse from Sec 3 with very confuing notations.  For example:\n\n1. L.102 frame-wise encoder-decoder, the encoder is the feature extractor and what is the decoder? why is it called a decoder?Also what is the value for dim $C$ used in exps? The reviewer failed to find such information.\n\n2. Likewise, L. 107 $P_i^m = \\phi_m(Q_l, F_d)$, what does $m$ indicate? and why the underscripts are inconsistent (m, l, d) in the equation, and it makes the reviewer guess about the meaning. The definitions of variable shall always be around where the variables are first introduced in text.\n\nThe authors should consider explain these notations for example with all variables at the same layer, i.e., a fixed underscript for all varibles to faciliate a clear understanding.\n\n\nThe current rating is boardline reject given the above concerns but the reviewer would consider raise the rating if the authors could address the issues.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for acknowledging our contributions and below is our response regarding the reviewer’s concerns.\n\n**1.For Weakness**:\n>However, given the design of voting between frame-wise and segment-wise predictions, one expected advantage of having access to segment prediction is to reduce the over-segmentation issue, yet the effect of such voting scheme between frame-wise and segment-wise predictions did not achieve the expected effect. On most of the metrics, the propose approach did not manage to outperform existing SOTA approaches, for example DiffAct, especooally on the segmental metrics.\n\n**Response**: \n\n We apologize for not providing a clear note on how to compare our performance with others, despite having separated the methods into single-stage and multi-stage in Table 6. Since our work focuses on the efficiency aspect of TAS, it is somewhat unfair to compare the single-stage (i.e., BaFormer) and multi-stage (i.e., DiffAct) models on segmental metrics. That is why we categorize all methods into multi-stage and single-stage groups, as shown in Table 6. Specifically, multi-stage models involve many stages of refinement, which are time-consuming. Our single-stage model is designed to be more efficient. Although our model shows a 0.5% decrease in F1@10, we improve accuracy by 0.8% on the 50Salads dataset, and notably, our model requires only **6%** of the inference time compared to DiffAct. DiffAct, based on a diffusion model, includes 25 inference steps, significantly increasing the time required to refine segmental metrics. When comparing models with similar inference times, DiffAct achieves **64.9%** on F1@10 and **88.6%** accuracy, while our method achieves **89.3%** on F1@10 and **89.5%** accuracy, demonstrating superior segmental metrics of our model. Further details can be found in Table 11 of the supplementary. \n\nWe can extend our methods to multi-stage to improve the segmental metrics further; however, experimental results indicate that this increases processing time, which contradicts our primary objective in this paper—to design an efficient TAS model, as acknowledged by the reviewer. To demonstrate the effect of our model, we also extended it into a multi-stage framework and modified the visual features for accuracy 91.2% and 90.8%, which are superior to DiffAct.\n\nTo fairly assess the effectiveness of the voting mechanism in reducing the over-segmentation issue, we need to compare it within single-stage models. These models all include one stage with post-processing, and comparisons should focus on different processing techniques with similar backbones. For instance, compared to UVAST with FIFA, our method demonstrates superior performance in segmental metrics (88.9% vs. 88.3%), accuracy (89.5% vs. 84.5%), and faster inference (0.139s vs. 1.765s), which result from the query-based model design. **This demonstrates that our approach has already outperformed existing SOTA approaches for the one-stage model.**\n\n**2. For weakness of confusing notation in Sec 3**\n\n(**About decoder**) Sorry for the confusion. **It should be noted that the encoder is frozen in all previous work for fair comparison**, mentioned in the experiment of the paper. Thus, the decoder aims to further process features from the encoder into high-level information, making them more adaptable to the TAS task through training. We call it a decoder to distinguish it from the frozen feature encoder/extractor. The decoder architecture can be any network adopting from the existing work such as MSTCN and Asformer.\n\n($C$ **in exps**) In the model, all instances of $C$ are set to 64, which is the feature dimension shared by different features. The value can be found in L217 of the experiment settings with \"an output dimension of 64\". We will clarify this by changing the phrase to \"an output dimension $C$ of 64\".\n\n(**Notation clarity**) Thanks for the suggestion and we will add improve the clarification and eliminate the notation inconsistency in the revision. Specifically, $\\( \\textbf{P}_i^m = \\varphi_m(\\textbf{Q}_l, \\textbf{F}_d) \\) $ represents the \\(i\\)-th mask features. Here, $m $ is the abbreviation of \"mask\".\n\nWe have reviewed all the notations and will use superscripts to represent the type of the variable and subscripts to represent the index. We will update the following notations:\n\n|        Old  |  New | Meaning |\n|:--------:|:----------------:|:----:|\n | $ \\textbf{F}_e, \\textbf{F}_d   $  |  $ \\textbf{F}^e, \\textbf{F}^d $| the output features of **e**ncoder or **d**ecoder |\n |  $ \\varphi_c, \\varphi_m,\\varphi_b $ |  $ \\varphi^c, \\varphi^m,\\varphi^b $ | heads for predicting query **c**lass, query **m**ask, **b**oundary  |\n\nFurthermore, for better understanding existing notations, we will add the following descriptions:\n|     Notation  |  Meaning |\n|:--------:|:----------------:|\n|$ \\textbf{Q}_l $ | instance query embedding from $l^{\\text{th}}$ layer |\n| $ \\textbf{P}_i^m $| Query masks $\\textbf{P}^m$ from $i^{\\text{th}}$ layer (the $m$ represents **m**ask.) |  \n| $\\textbf{P}_i^b $| Boundary probability $\\textbf{P}^b$ from $i^{\\text{th}}$ layer (the $b$ represent **b**oundary.)  |\n| $\\textbf{y}_i^m $| binary mask $\\textbf{y}^m$ from the $i^{\\text{th}}$ query (the m represents **m**ask) |\n\n**3. For Question 1**\n\n(**Advantage of extra branch**). As our approach is based on voting to yield segmentation results, the extra branch for boundary generation is essential for our method. The extra branch can generate high-quality boundaries without much additional effort (nearly 5% Flops assumption). It can bring a 2.3% improvement in accuracy and a 2.1% improvement in Edit score, compared to a direct summary of all query predictions without an extra boundary prediction branch.\n\n**4. For Question 2**\n\nRefer to the answer of above part \"2. For weakness of confusing notation in Sec 3\" .", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CqbMMTFPVq", "reviewer": "Reviewer_KZs3", "summary": "The authors introduce BaFormer, an innovative single-stage boundary-aware Transformer network designed for temporal action segmentation (TAS). This method employs instance queries for instance segmentation and a global query for class-agnostic boundary prediction, achieving substantial reductions in computational costs while preserving or enhancing accuracy relative to state-of-the-art approaches. Experiments across multiple datasets highlight BaFormer's efficiency and effectiveness, demonstrating its capability to achieve competitive results with significantly lower running time and resource usage.", "questions": "1. How could the variation in the number of query tokens impact the results of the proposed method?\n\n\n2. Can the authors provide a detailed comparison of the different loss functions used in the proposed method?\n\n3. Could you provide examples of baseline methods that should be included in the qualitative comparison?\n\n4. Can you provide examples of how changes in hyperparameters could affect the performance of the proposed method?", "limitations": "No, the authors mentioned that the limitations are included in the supplementary however there is no supplementary materials for this work.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed approach is interesting and show good performances on various of datasets.\n\n\n2. The method is clearly described and easy to understand. The whole paper is well-written,\n\n\n3. Efficient temporal action segmentation is an important research direction.", "weaknesses": "1. Lack of the ablation studies. The authors are encouraged to add more ablation studies, e.g., regarding different number of query tokens. Apart from those ablations, the ablation of the loss function is also interesting since there are so many losses incorporated in the proposed method.\n\n\n2. Lack of qualitative results.  The authors are also encouraged to provide some qualitative results or TSNE figure to show more insights regarding the efficacy of the proposed method compared with some baselines.\n\n\n3. Since the authors mentioned that the proposed method is efficient, however when we have a look at Table 1, the proposed approach do not clearly show the benefits in terms of the model size compared with DiffAct, which is regarded as one weakness.\n\n4.  The sensitivity of the parameters is interesting to be analyzed, since the proposed method do preserve large amount of the hyper parameters.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the valuable suggestions to enrich the analysis of our model, most of which are included in the existing supplementary. It looks like you didn’t find the supplementary material -- we double-checked and it’s not a problem, and other reviewers haven’t had this issue either. Please inform us or contact the conference organizer/AC if there are any issues with locating or downloading the supplementary material. We refer to the location of the following question in our supplementary material.\n\n(**Weakness 1 and Question 1: more ablation studies and query tokens**)  We agree that more ablation studies help us to analyze the model further, and we include some of them in supplementary,  such as the number of query tokens, single/multi-level feature connections, auxiliary loss, and\nother factors.  Specifically, as for the number of query tokens, the experiment indicates that a slight increase in the number of queries can enhance the model’s ability without significantly raising computational costs. However, the model’s performance peaks at 100 queries, achieving the highest accuracy of 89.5% at a computational cost of 4.45G FLOPs (more details can refer to Table 2 of the supplementary material). \n\n\n(**Weakness 1 and Question 2: a detailed comparison of the different loss functions**) In supplementary Table 3, we present the performance results when the auxiliary loss is included under single or multiple features. Notably, applying auxiliary losses in multi-level features yields a substantial improvement in accuracy (2.1%), surpassing the 0.5% accuracy gain observed in single-level features. This suggests that auxiliary losses have a more pronounced effect on multi-level feature representations.\n\n(**Weakness 2 and Question 3: qualitative results**) Qualitative results, which can help us to analyze segmentation results, especially among the time dimension. So, in the supplementary material, we also provide additional qualitative results in Figure 2 to demonstrate the efficacy of our proposed query voting.  In the 50Salads dataset, we present enhanced visualizations for a more comprehensive analysis. The figure includes the outcomes of instance segmentation, the segmentation results without boundary integration,  and the ground truth. \n\n(**Weakness 3: model size compared with DiffAct**)We apologize for the concerns and recognize that we should clarify that efficiency refers to shorter running time. Thus, despite DiffAct's smaller size, its running time is longer than ours. DiffAct is a multi-stage method based on the diffusion model. It is built on a model with a small size but requires 25 iterations to refine the results, significantly increasing the running time. By contrast, our BaFormer requires only 6% of the time used by DiffAct and obtains a comparable performance. Directly comparing single-stage methods with multi-stage ones is somewhat unfair, we therefore divide them into two groups in Table 6. Alternatively, we can adjust the models to have similar running times for a fair comparison, we present the results in Table 7. Our method shows superior performance, achieving an accuracy of 89.5% and F1@10 of 89.3, compared to DiffAct's accuracy of 74.2% and F1@10 of 48.3 (more experimental results with different steps for DiffAct are shown in Table 4 in the supplementary material). Furthermore, we can extend our method to a multi-stage one to increase the accuracy to 90.5%, which surpasses the original DiffAct. However, in this paper, we focus on improving the performance of single-stage methods rather than relying on multi-stage methods.\n\n(**Weakness 4 and Question 4: more analysis of hyperparameters**) The supplementary material includes an analysis of hyperparameters, such as the number of the query tokens(Table 2), the connection between the frame-wise and transformer module (Table 3),  the transformer decoder layer(Table 1),  the single or multiple features (Table 3) and so on.\n\n(**Limitation**) The limitation is also discussed in the supplementary. Our proposed approach is under the query-based Transformer framework, which is well-known for its slow training convergence and data thirty. We observe that training query-based Transformer converges slower than training frame-wise approaches. For the discontinuous binary mask predictions, we assume that it may also be attributed to the limited data of action segmentation benchmarks. We hope that these factors can inspire future work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vn4ia1kZ7E", "reviewer": "Reviewer_KTbs", "summary": "The paper introduces BaFormer, a novel Transformer network designed to improve the efficiency of Temporal Action Segmentation (TAS) while maintaining high performance. BaFormer tokenizes video segments into instance tokens and utilizes instance queries for segmentation along with a global query for boundary prediction, resulting in continuous segment proposals. The model employs a voting strategy during inference to classify segments based on instance segmentation, significantly reducing computational costs. Experiments on popular TAS benchmarks demonstrate BaFormer's superior efficiency with comparable or better accuracy than state-of-the-art methods like DiffAct.", "questions": "Please mainly see the weaknesses section for my problems.", "limitations": "the authors have adequately addressed the limitations", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "In general, the proposed method is somehow simple but effective, and I am satisfied with the paper. The proposed Beformer utilizes a simple structure but can achieve performance gain and inference speed gain over previous methods. \n\nThe proposed method also has detailed ablation studies, which can verify the effectiveness of the proposed BeFormer. Though BeFormer is a single-stage method, it performs comparable to the most recent two-stage methods.", "weaknesses": "However, there are some small concerns about the paper. \n\n1. Despite the amazing performance, I think the novelty is limited. The author utilizes existing frameworks, including existing frame encoder and ASFormer encoder, also, the decoder is a common query-based decoder. The main novelty exists in the boundary-aware query voting part. However, it is a post-processing process. Thus in general the novelty is minor.\n\n2. In Table 1 (teaser), the author claims that BaFormer with SSTCN and ASFormer backbones performs better than the original module with less running time. However, in Table 7, they have more FLOPs than the original methods but with less inference speed. Does the extra running time come from the NMS post-processing? If so, I think it is somehow unfair here. Maybe the author could list the modeling forward time and the post-processing time for a better and fair comparison. Otherwise, it is somehow confusing here.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for investing time in reviewing our work and acknowledging our contributions.\n\n**1. For Weakness 1**\n\n(**utilizes existing frameworks**) We understand the reviewer's concerns regarding our use of existing frameworks; however, we employ them solely to achieve the desired functionality. An important aspect we need to clarify is that we focus on how to adopt a new pipeline, i.e., query-based pipeline, into efficient TAS rather than some specific network design.  To achieve more efficient TAS, we experimentally find that query-based processing can reduce dense information, thus reducing FLOPs. Therefore, we adopt query-based prediction from a DETR-style model to improve the running time. This lets us establish a new paradigm in TAS, different from the frame prediction in previous work. Our main effort is how to adopt the DETR-style model into TAS to achieve the expected performance, rather than to design some specific network structures. Specifically, we focus on introducing functional modules with effective organization, i.e., frame-wise encoder and decoder, transformer decoder. These modules can be implemented with any network. However, in our paper, as acknowledged by the reviewer, we employ existing frameworks to achieve these functional modules. This approach allows us to: 1) fairly compare with previous methods by sharing the same backbone, and 2) verify the effectiveness of our pipeline, which relies on the new paradigm rather than network design.\n\nFurthermore, although these existing frameworks have been successful in previous works, they may not be effective in our new pipeline, which is organized with different functional modules.  Therefore, in terms of TAS tasks, we have made numerous efforts to optimize our approach. These efforts include experimenting with different matching strategies, establishing connections between frame decoders and transformer decoders, and implementing auxiliary loss functions. We analyze these aspects in detail in the main paper and supplementary materials and hence provide insights for designing a superior pipeline. \n\nIt is worth mentioning that there are many impactful works to adopt the DETR-style model into different fields as what we do in TAS tasks, e.g., TrackFormer[1] for multi-object tracking, MaskFormer[2] for image segmentation and CLTR[3] for crowd localization. However, our work is the first such exploration for TAS.\n\n[1] T. Meinhardt, A. Kirillov, L. Leal-Taixe, and C. Feichtenhofer. Trackformer: Multi-objecttracking with transformers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8844–8854, 2022.\n\n[2] B. Cheng, A. Schwing, and A. Kirillov. Per-pixel classification is not all you need for semantic segmentation. Advances in Neural Information Processing Systems, 34:17864–17875, 2021.\n\n[3] D. Liang, W. Xu, and X. Bai. An end-to-end transformer model for crowd localization. In European Conference on Computer Vision, pages 38–54. Springer, 2022. \n\n**2 .For Weaknees 2**\n\n(**Running Time**) Thank you for the suggestion to include detailed results to improve our paper. Based on the reviewer's suggestion, we have added the table below. And find that the original module's additional running time is due to the NMS post-processing. To ensure a fair comparison, we have listed both the forward time and the post-processing time below. Thanks again for the reminder and we will update the manuscript.\n\n|Method | Total time(s) | Backbone time(s) | Post-processing time(s) |\n|:--------:|:----------------:|:----:|:--------:|\n|SSTCN| 0.080 |0.031 | 0.049|\n|BaFormer(SSTCN) | 0.074 | 0.040 | 0.034|\n|ASFormer(Encoder) | 0.158 |0.092 | 0.066 |\n|BaFormer(ASFormer Encoder) |0.139 |0.105 | 0.034 |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper received mixed reviews, with two borderline rejects, one borderline accept and one accept.  All the reviewers appreciate the methodology and effectiveness, while there are some concerns with respect to technical clarifications. 

The AC has carefully reviewed the paper, author response and discussion and sees value in the work to the NeurIPS community and therefore recommends acceptance.  The authors are recommended to revise their work based on the discussions and clarifications with the reviewers.
Decision: Accept (poster)