Alignment-guided Temporal Attention for Video Action Recognition
Authors: Yizhou Zhao
Zhenyang Li
Xun Guo
Yan Lu
Venue: NeurIPS 2022 Accept (2022)
Keywords: Temporal Attention
Action Recognition
Video Learning
Mutual Information

Abstract: Temporal modeling is crucial for various video learning tasks. Most recent approaches employ either factorized (2D+1D) or joint (3D) spatial-temporal operations to extract temporal contexts from the input frames. While the former is more efficient in computation, the latter often obtains better performance. In this paper, we attribute this to a dilemma between the sufficiency and the efficiency of interactions among various positions in different frames. These interactions affect the extraction of task-relevant information shared among frames. To resolve this issue, we prove that frame-by-frame alignments have the potential to increase the mutual information between frame representations, thereby including more task-relevant information to boost effectiveness. Then we propose Alignment-guided Temporal Attention (ATA) to extend 1-dimensional temporal attention with parameter-free patch-level alignments between neighboring frames. It can act as a general plug-in for image backbones to conduct the action recognition task without any model-specific design. Extensive experiments on multiple benchmarks demonstrate the superiority and generality of our module.

Alignment-guided Temporal Attention for Video Action Recognition
Yizhou Zhao
Zhenyang Li Xun Guo
Yan Lu
Carnegiee Mellon University
TTinghua University *Micrsooft Research Asia y yizhouz@andrew cmu. edu, 1izy200mails tsinghua. edu. cn {xunguo, yanlu)@microsoff com
Abstract
Temporal modeling is crucial for various video learning tasks. Most recent ap- proaches employ either factorized (2D+ID) or joint (3D) spatial-temporal oper- ations to extract temporal contexts from the input frames. While the former more efficient in computation, the latter often obtains better performance, In this paper, we attribute this dilemma between the sufficiency and the efficiency of interactions among various positions in different frames. These interactions affect the extraction task-relevant information shared among frames. To resolve this issue, we prove that frame-by-frand by-mrame alignments have the potential to increase the mutual information between frame representations, thereby including more task-relevant information to boost effectivvness. Then propose Alignment- guided Temporal Attention (ATA) to extend I-dimensional temporal attention with parameter-free free patch-level alignments between neighboring frames. It can act as a general plug-in for image backbones to conduct the action recognition task without any model-specifee design. Extensive experiments multiple benchmarks demonstrate the superiority and generality our module.
Introduction
Unlike images, videos include a wealth temporal information due to common and distinct patterns in nearby frames. In images, the whole scene is static. In videos, however, the background, the foreground, and each part of them can have different degrees of movement. These differences necessitate temporal modeling in video learning tasks.
With respect to this, recent literature mainly focuses on two branches of modelings, namely, factorized (2D+1D) spatial-temporal operations (49 [5 34] and joint (3D) spatial-temporal operations [42] 37 32). The former considers staggering or dividing spatial operations with temporal ones, such separable 3D convolutions and factorized spatoal-temporal attention. Since the two groups of operations are relatively independent, this paradigm leverages indirect interactions for features at different space-time locations. Enjoying the efficiency from fewer participants for each operation, often suffers from less sufficiency cross cross-frame frame cross-location interactions. latter extends 2-dimensional operations their 3-dimensional varieties, thereby jointly interacting features from neighboring areas as well as frames, e.g., 2D convolutions vs. 3D convolutions and spatial attention vs. joint spatial-temporal attention. This line of work explores view compared with the former one. bringing more direct and adequate interactions at the cost of dramatically increased complexity. As result, conflict seemingly exists between the sufficiency and the efficiency of interactions among various positions in different frames.
contrast, the
global
Work done during internship Microsoft Research Asia.
i Equal contribution.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Input representations X1 & X2
Conventional Temporal Attention
relevatl information
Task & representations shared information
Representations X & X2 shared information
Alinment-guided AttporalAttennion
Attention (Ours)
Increase &x2 mutualiiformation
fz
(a) Conventional
(b) Alignment guided (Ours)
Spapial-tempora Operation
spatial-tempont Operation
Figure 1: Information diagram illustrating our Figure Comparison between conventional motivation. I stands for task-relevant information, Temporal Attention and Alignment-guide and X1. X2 denote representations of adjacent in- Temporal Attention (ATA). Participants of each put frames f1, f2-
operation are connected with lines.
To address this conflict, revisit both schemes from the perspective of information theory. With the help of more direct interactions, joint spatial-temporal operations share more mutual information between neighboring frame representations than their factorized counterparts. This potentially increases task-relevant information extracted by the model, benefiting its performance. Nevertheless, for joint spatial-temport modeling, the ability to exploit mutual information from consecutive frames depends on the kernel size of convolution-bast To reach the theoretical upper bound of mutual information shared, completely global interactions are needed for this paradigm, which is not realistic. In general cases, these conventional schemes collect inadequate task-relevant information as illustrated in Fig. T(a..
operations or the window size of attention- ones.
Alternatively, we propose to directly interact between the most similar parts of adjacent frames, instead of among regions at nearby spatial locations. In other words, we guide the spatial-temporal operation with alignments as shown in Fig. 2 We theoretically prove that this guidance increases mutual information of neighboring frame representations in Fig. [b) and practically implement this concept as our Aligument- figumeen-guuiedd guided Temporal Attention (ATA). Upon traditional Temporal Attention, it has no extra parameters and low additional computation cost. For alignment, we match patches of adjacent frames with the Kuhn-Munkres Algorithm (KMA) [4] based on cosine similarity. After temporal attention through the aligned route, de-align patches to keep their original spatial order.
Our contributions can be summed up as follows.
1. We discuss the problems of factorized (2D+ID) and joint (3D) spatial-temporal operations and propose Alignment-gunl guided Temporal Attention (ATA) for temporal modeling. Taking advantages of both, ATA achieves strong performance while maintaining low computational overheads.
2. We theoretically prove that the frame-by-frame alignment increases mutual information between neighboring frame representations, thereby potentially including more task-relevant information.
3. Through qualitative and quantitative experiments, we demonstrate that our ATA is superior to conventional temporal modeling methods by providing more effective cross-frame cross-location interactions. Implementations on video frameworks and image backbones (TimeSformer [3] CycleMLP 66, and ConvNeXt (3T) show its potential as a general glug-and-play module to bridge from image recognition to video action recognition.
2
Related Work
Visual learning architectures. Due the high generalization of ViT [10], numerous Transformer- based image backbones recently extend their work range to video. TimeSformer (3] presents four kinds of factorized space-time implements utilizing the standard ViT structure. VTN 33] uses temporal attention model to aggregate information from all video clips. MVIT [I]] creates multi- scale architecture through integrating pooling and downscale methods, which have both been used in image backbones. ViVIT 2 creates three variants that focus on factorized space-time attention. It leverages tubelet embeddings to extend frame patch representations into teeporllt volume representations.

---

(a) Joint (3D) spatial-temporat Operation
z
Spatial-temporal Operation
z
(b) Factorized (2D+1D) seatialltemporal Operation
-
Temporal
Spatial Operation
MLP
Z
Operation
(c) Alignment-guides spatial-temporal Operation (Ours)
z
Frame-wise Alignment
Temporal Operation
De- alignment
Spatial Operation
MLP
Z
Figure Comparison among spatial-temporal operations in terms of model structures.
Video Action recognition. Early action recognition research 266
43 35
focuses on hand-
crafted features. With the advancement deep neural networks, schemes adopting learnable video representations gradually surpass traditional methods based DT
44 and iDT (43] To be specific, AlexNet [23] generates numerous important video architectures, which include 2D/31)
CNN 1 28 in two-stream models and its variants with LSTM/GRU/RNN, [9] 38 51] Recently
Transformers- based ViT (10) spawns a series of video learning architectures on the basis of the attention mechanism, such as VTN 33], TimeSformer
3), ViVIT 2 and Video Swin [32]
Temporal correspondences. In comparison image recognition. motion representation is the most difficult aspect of video action recognition [25] 20 36 18 48] Recent alignment methods have primarily focused on supervised temporal correspondences. For representing temporal relation- is proposed to establish the displacements in pixels between successive frames. Patch-level interconnections can reduce the memory cost of optical flow. For instance, (29) presents a typical descriptor model called SIFT flow and the SIFT features are being used for matching patches. Local features such as HOG [24 i and MBH [8] can be used to describe statistical features well. The correlation of patches or other inter-frame components, in particular, has positive effects on models learning visual representations, according 41).
ships, optical flow [16 17 27 39
Method
For video learning tasks, we consider their inputs a combination spatial and temporal information. The former is introduced by sequential static frames, while the latter originates from their potential connections. These two parts are usually tackled jointly convolution- in attention-based approaches [30], with their per-block model structures roughly demonstrated in
methods (5) as well as
Fig. B]). [). i.e., joint (3D) spatial-temporal operations. We also notice attempts 49 2 separating spatial operations from temporal ones in Fig. 3b) known factorized ID) spatial-temporal operations. For either scheme, the ability to extract mutual information from consecutive frames is restricted by the kernel/window size of its temporal operation. To mitigate this, we propose to conduct frame-wist alignment before each temporal operation and de-alignment after that as in Fig. 3)c.
3.1 Problem definition
To begin with, define alignment generally. Given two neighboring frames X X e HWCC HW
HW the alignment function of X and X is with their patch embeddings { Ji=i {x {
x
xt) = arg max {x [X(-1__0__ ,(X*))]
where is criterion function and p HW x + T HC is projection. The de-alignment
function a -1 defined as the reverse process of alignment
a (a (x (x)-1,x))) x
(2)
3

---

Specifically, we can have spatial rearrangement series of projection
Rearrange (x*)
R X = X ( Hw
(3)
where the rearrangement matrix
R = ( ij
1-1,t]
E
[0, 1]
HW
HW
HW
(4)
s.t.
1-1,
1,
1-1,4
bd
1=1
1=1
and cosine similarity as the criterion X
X Cosine
(x-1,-)== Σ
2
(5)
In this case, the rearrangement can be regarded as a patch-wist match between X and X and the alignment should be a perfect matching scheme. As a result, the alignment matrix is one of the rearrangement matrices satisfying
A arg max (X Cosine (x*-1, Rx x')
(6)
and the de-alignment matrix is simply its transpose
D= AT
3.2 Proof and analysis
Mutual information of two random variables measures the information shared between them. It represents the degree of reduction in uncertainty of one variable given the other. Traditional ID temporal operations interact features of consecutive frames at certain spatial position. Since these features might diverge greatly, their irrelevances can lead to uncertainty, decreasing mutual information collected by the model. We will prove that the mutual information of adjacent frames increases after alignment, which also indicates that the similarity of adjacent frames rises.
We define the representations of two adjacent frames as random variables X and X Within the same video clip, they obey the same distribution. Thus their mutual information can be calculated as
MI(X'-1,X') H(X") H(X* |
Eq. expresses the mutual information between the feature Xt-1 of the (t - 1)th frame and the feature X of the thh frame. The definitions of information entropy H(X²) (t E {1...., T)) and conditional entropy H(X H(X+)x(-1) are as follows.
HW
H(X*) = - Σ
p (x logp (x¹)
(9)
1=1
x
Σ pla - H(X* xt-1 = -1
(10)
Σ p(a 1 Σ pla 1 log(p(()) bzt
where 2²-1 E Ft-1 and x E Ft. Ft-1 and F denote the feature sets of all patches in the (t frame and the 4t frame, respectively. p(x:-1) the probability of occurrence of random variable Xt-1, p(az ===²) the conditional probability of occurrence of random variable X" under the condition determined by random variable X p(z), x 11) is the joint probability that random variables t and X satisfy certain conditions at the same time.
1)th

---

Input representations X₃ & 22
Tesk-relevant information
Task & representations shared information
Representations X₃ & X2 shared information
fz (a) Factorized (2D+1D) Spatial-temporal Operation Sufficiency Efficiency
f1
f1
f2
f1
f2
(b) Alignment guided (Ours) tpattal-temporal Operation Sufficiency Efficiency
(c) Joint (3D)
Spatial-temporal Operation Sufficiency Efficiency
Figure 4: Comparison of spatial-temporal operations terms information diagrams. T represents task-relevant information, and X1, X2 are representations of adjacent input frames fı, f2-
Given patch embeddings a certain frame, the probability of a certain patch embedding equaling a certain vector is determined, i.e.,
1
P (2 -
(11)
HW where z²j indicates the feature of the jrh patch in the 4t frame before the alignment, and & means the feature of the jth patch in the tt frame after the alignment.
Hence Eq. (10) can simplified to
H(X* x = HW
Σ p(a) | log(p(o) at at
(12)
The rearrangement-based alignment does change the feature contents of the patch, thus the information entropy does not change after alignment. As a result, the mutual information of the features of two adjacent frames after the alignment is
MI(X MI(X*=1,X') X') = H(X*) H(X* | H(X*) H(X' x
(13)
We notice that the difference between Eq. 8 and Eq. 13) lies in the different conditional entropy
H(X' xx=1) and H(X' xt=1). With H(X* X expressed in Eq. 12 H(X*
x²-1) can
be similarly written as
H(X x =- HW Σ
p( log(pp() x x
(14)
We assume that two patches that are close to each other have a greater probability to be similar than the probability to be dissimilar. If we align the patches according to the Eq. (6). more patches with high similarity have the chance to be put in close positions. In this cireumstance, patches with high similarity will have greater probability of appearing temporally close positions in adjacent frames. On the one hand, when the positions of and are close, the probability similar features appearing in such close locations becomes higher. On the other hand, when the positions of i and j are far away, the dissimilar features are more likely to occur at such a distance too. In general, the conditional probability p( -1 will become greater than p(r5 x 1) Consequently, Eq. (10) is greater
than Eq. 114]
H(X* X
X(-1)
(15)
As the conditional entropy is negatively correlated with the mutual information, the conditional en- tropy decreases after alignment, and the mutual information MI(X X²) increases correspondingly.
MI(X > MI(XI-1,X1)
(16)
5

---

Rearrange
Aligned x
Original x
Similarity 5 st
One-hot Binary Mask A
Keep
Figure 5: Alignment with Kuhn-Munkres Algorithm (KMA). Redder cells indicate higher similar- ities for paired features, while bluer ones mean the opposite.
The performance of a model is related to how much task-relevant information it collects from its input [45). In video learning tasks, this task-relevant information is subset of frame representations, which are information extracted from input frames. For simplicity, we present an instance of two neighboring frames demonstrated in Fig. Under the same model capacity, information extracted from each frame is approximately the same amount for each model. Nevertheless, the situation is different in task-relevant information. When modeling with factorized (2D+1D) spatial-tempont operations in Fig. [a), the shared information of frame representations is rather small due to the misaligmment of patches participating
single temporal operation. With a more global spatial- temporal view, joint (3D) operations in Fig. 4c) tend to increase this shared information. It also results in more task-relevant information extracted but is still restricted by the kernel/window size. Since enlarging kernel/window size cubically raises the complexity, this line of work is often less efficient. As for our alignment- guided spatial-temporal operation in Fig. 4(b), can increase the mutual information shared by neighboring frame representations with simple operations, thereby having the potential to collect more task-relevant information at a low cost. This helps with both the sufficiency and the efficiency of cross- cross-frame cross-location interactions and leads to better performance.
3.3 Solution
In order solve the problem discussed, the key is obtain alignment matrix A. To this end, potential algorithms are supposed to produce a one-on-one matching matrix according to the given criterion. This can be simplified classical bipartite matching problem, which is efficient solve through Kuhn-Munkres Algorithm (KMA) 4
Alignment. Given the cosine similarity criterion in Eq. 5), the alignment procedure with KMA is illustrated in Fig. 5 Firstly, we calculate the similarity matrix of two adjacent frames X and X
1t
(17)
Based on this, KMA produces a one-hot binary matrix matching patch embeddings from the previous frame and patches from the current
A = KMA (5t-1.)
(18)
Then the spatial order of the previous frame features is kept, and that of the current is rearranged according to Eq. (3). This process is applied to every neighboring frame pair, thereby producing an
aligned route for 1D temporal operations. Since the computational complexity of KMA is O(N3) 119
alignment with KMA has a complexity of O H333).
De-alignment. After aligning frame representations with rearrangement, the spatial structures of them are modified. To keep the original location- information and facilitate subsequent spatial operations, a recovery with de-alignment as in Eq. (7) is thus needed. As result, the alignment, the temporal operation, and the de-alignment as whole form temporal interaction through a 3D path, within which each point on the path is similar to the next one.

---

Table I: Comparison to the state-of-the-art on Kinetics-400. views means spatial crops with y temporal clips. We report the inference cost with single view. RED/BLUE indicate SOTA/the second best. IN stands for officially released ImageNet 23 pretrained models. The same below.
Method
Pretrain
Frames
Top-1 Acc
Top-5 Acc
Views
FLOPss (G)
Params (M)
VIT-B-VTN
33
IN-21K
16
79.8
94.2
1
4218
114
VidTr-L 50
IN-21K
32
79.1
93.9
3 3 10
351
ViVIT-L
IN-21K
16
81.3
94.7
3
3992
310.8
TimeSformer
IN-21K
8
78.0
93.7
3
196
121.4
l'meeSormer-L.
IN-21K
96
80.7
94.7
3
2380
121.4
TimeSformer-In HR
IN-2IK
16
94.4
1703
121.4
MViTv1-B
16
93.5
1
70.5
36.6
MViTv1-B
32
80.2
94.4
1
170
36.6
MViTv1-B
64
81.2
95.1
3 3X3
455
36.6
Mformer-B B
34
IN-21K
16
79.7
94.2
3 X 10
369.5
109.1
Mformer-I
34
IN-21K
32
80.2
94.8
3 X 10
1185.1
109.1
Mformer-Hie HR
34
IN-21K
16
81.1
95.2
3 X LO
958.8
381.9
ATA (Ours)
IN-21K
32
81.4
95.5
3 X
792.9
121.8
ATA (Ours)
IN-21K
32
81.9
95.5
3 X
792.9
121.8
Table 2: Comparison to the state-of-the-art on Something-Sommething V2. IN-2IK+K-400 denotes pretraining the video architecture K-400 based on ImageNet [23] pretrained image models.
Method
Pretrain
Frames
Top-I Acc
Top-5 Ace
Views
FLOPs
Params (M)
ViVIT-L
IN-21K+K-400
16
65.4
89.8
903
352.1
TimeSformer
IN-21K
59.5
74.9
3
196.7
121.4
TimeSformer-L.
IN-21K
62.4
81.0
3
2380
121.4
TimeSformer-IFe
IN-21K
62.2
78.0
3
121.4
MViTvI-B
K-400
64.7
89.2
3
70.5
36.6
MViTv1-B
K-400
32
67.1
90.8
3 1
170
36.6
MViTvI-B
K-400
64
67.7
90.9
3 I
455
36.6
Mformer-B B
34
IN-21K+K-400
16
66.5
90.1
3 1
369.5
109.1
Mformer-1 L
34
IN-21K+K K+K-400
32
68.1
91.2
3 1
1185.1
109.1
Mformer-HR HR
IN-21K+K-400
16
67.1
90.6
3 X
958.8
381.9
ATA (Ours)
IN-2IK
32
67.0
91.0
792.9
121.8
ATA (Ours)
IN-21K
32
67.1
90.8
X
792.9
121.8
4 Experiments
4.1 Experimental Setting
Benchmarks.
We employ two widely used video action recognition datasets, i.e., Kinetics-400 (K-
400) [22] and Sooeehing-Somtthing V2 (SSv2) 14,, in our experiments. Kinetics-400 contains 240k training videos and 30k validation videos in 400 classes of human actions. consists of 168.9K training videos and 24.7K validation videos for 174 classes. We provide the top-1 and top-5 accuracy the validation sets, the inference complexity measured with FLOPs, and the model capacity in terms of the number of parameters.
V2
Implementation details. We use TineSformer 3 with the officially released model pretrained on
ImageNet-211 (23 as our baseline. After inserting our ATA to each Transformer block, the model is then finetuned on video datasets with standard augmentations as in [12. We adopt SGD to optimize our network for 30 epochs with mini-batch size of 64. The initial learning rate is set to 0.005 with 0.1x decays the 2 and 27th epochs. All patch embeddings of le - 4, while the class tokens and the positional embeddings used no weight decay. In the main experiments compared with the state-of-the both K-400 and SSv2. And in the ablation study, we provide the results of 8-frame input and 3x testing views K-400. The resolution of 224 x 224 is used throughout all the experiments.
applied with weight decay
methods, provide the results of 32-frame input

---

Table 3: Ablation study of temporal modeling and de-alignment on Kinetics-400. For temporal modeling, averaging refers to computing the mean of all spatial-temporal features, and attention means factorized temporal attention. The same below.
Image Backbone
Pretrain
Temporal Modeling
De De-alignment
Top-1 Acc
Top-5 Acc
FLOPs (G)
Params (M)
Averaging
75.3
Attention 3
76.8
93.1
122.4
102.8
CycleMI.-B33
IN-IK
ATA (Ours)
X
72.7
90.9
122.4
102.8
ATA (Ours)
V
77.7
93.5
122.4
102.8
Averaging 46
79.0
94.0
123.0
88.0
Attention 3
80.1
94.8
198.0
140.5
ConyNeXt-Bass
IN-22K
ATA (Ours)
x
76.0
92.7
198.0
140.5
ATA (Ours)
80.5
94.8
198.0
140.5
Averaging
76.0
92.6
140.3
86.2
Attention 3
93.7
201.8
121.8
VIT-Base 10
IN-2IK
ATA (Ours)
x 79.3
94.3
201.8
121.8
ATA (Ours)
/
79.6
94.3
201.8
121.8
4.2 Comparison Results
Kinetics-400. Table presents comparison to the state-of-the-arr results K-400. As our motivation demonstrate the effectiveness of alignment without bells and whistles, we compare our model only with the approaches based on VIT to reduce the impact of architectural inconsistency. All these methods are pretrained on ImageNet-211 except for MVIT [T], which trained from scratch. Results show that our method outperforms most of the existing ViT-based approaches. It is noteworthy that our method has the same factorized spatial-temporal structure but a different form of temporal modeling with TimeSformer [3]. Using fewer input frames and lower resolution, ATA even surpasses Tfomerr and Time
which can fully demonstrate its effectiveness. Compared with Motionformer [34], which explores motion trajectory in temporal attention, method achieves better performance with much lower complexity. Our approach can achieve better performance using pretrained models larger datasets, e.g., JFT-300M.
V2. Table compares our approach with the state-of-the- methods on SSv2. Different from K-400, SSv2 consists of vidcos with high temporal reasoning. Therefore, pretraining video data would very helpful. That may be the reason why our method shows lower performance than Motionformer-L. at the same input setting. It is noteworthy that our method achieves comparable results MViT-B with 22-frame input. Considering its 3D structure and pretraining on K-400, our method demonstrates stronger ability in temporal modeling.
Homething-Somettiee
4.3 Ablation study
Generality various image backbones. To investigate the generality of ATA, we add three kinds of temporal modeling three distinct image architectures (MLP-based, Conv-based, and attention- based) in Table 3, Note that we only consider ATA with de-alignment in this ablation. Based on each image backbone, temporal attention obtains improvements of 1.9%, 0.9%, and 2.0% accordingly compared with averaging, with the help of interactions among frames. Since the attention mechanism explicitly considers the correlation of frame-wise features, its outputs will contain more mutual information than the inputs. Moreover, with alignment and de-alignment, ATA achieves extra gains of 0.9%, 0.4%, and 1.6%. This is the result of better choices of attention participants, which further increases the mutual information among frames. Consequently, more mutual information gathered leads to a better understanding of video sequences, thereby bringing higher performance.
Effectiveness of de-alignment. We compare our ATA with and without de-alignment in the last two lines of each architecture in Table 3 For models based on the same image backbones, see performance drops ranging from 0.3% to 5.0% in top-1 accuracy and from 0.0% to 2.6% in top-5 accuracy. This demonstrates the necessity of de-alignment. Moreover, we notice that models based on CycleMLP 6 (rows 1-2) and ConvNeXt [1 (rows 3-4) experience larger fall than that based on ViT TO (rows 5-6). Its potential reason might that CycleMLP 6 and ConvNeXt (31] rely heavier on local spatial features. These features will be confused by rearrangement-basee alignment, and thus damage the learning of spatial structures. Since ViT I0) adopts the attention mechanism, which is global for spatial features, its spatial modeling less affected.

---

Table 4: Comparison of Mutual Information on Kinetics-400. "None" denotes the result calculated right after patch embedding. Mutual information is averaged among all pairs of adjacent frames.
Image Backhone
Pretrain
Temporal Modeling
Kinetics-400
Something- V2
None
0.243
0.295
Averaging
46
1,355
0.869
ViT-Base 10
IN-21K
Attention
3
1.371
0.910
ATA (Ours)
1.373
1.290
Table 5: Ablation study of inserting locations on Kinetics-400. Block x-y means replacing the
original temporal attention with our ATA on corresponding encoder blocks.
Block 0-2 Black 3-5
Block 6-8 Block 9-11
Top-1 Acc
'Top-5 Ace
FLOPS (G)
Params (M)
78.0
93.7
201.8
121.8
79.3
94.4
201.8
121.8
79.6
94.4
201.8
121.8
79.3
94.2
201.8
121.8
79.2
94.2
201.8
121.8
79.6
94.3
201.8
121.8
Comparison of mutual information. As we theoretically proved, alignment increases mutual information between neighboring frame representations. With respect to this, practical results are in Table Clearly, we can see that mutual information is extremely low without temporal modeling, Simple averaging can add to mutual information by 1.112 on K-400 and 0.574 on SSv2, while attention increases by 1.128 and 0.615 respectively. On the basis of attention, our parameter-fre aligmment obtains even more growth. We also notice that the difference that alignment makes is more pronounced SSv2, possibly resulting from more dynamic temporal information this dataset. This whole trend is correspondent to the performance. As a result, alignment can indeed increase mutual information and include more task information, leading to soaring performance.
Impact inserting locations. We divide 12 blocks of our baseline into 4 stages, each containing blocks. Then we insert our ATA in each stage individually, as shown in Table 5 Generally, per-stage insertions of ATA rows 2-5 obtain 1.2% 1.6% gains in top-1 accuracy and 0.5% ~ 0.7% gains in top-5 accuracy compared with the baseline in the first row. These results are approximately the same as that in the row, which equips every block in the baseline with alignment and de-alignment. This shows that even partially inserting ATA helps increase mutual information for adjacent frames, thereby benefiting the performance. We also notice that the 3rd row achieves the best performance in the table. It suggests that the selection of features to align and de-align also makes difference, besides the operations themselves.
Limitation Discussion
As preliminary attempt toward the frame alignment problem, ATA is both parameter-fiv free and not differentiable, which might limit its upper-bound bound of performance. To this end, the process of matching features from adjacent frames can be potentially conducted by network, or it can be modeled by consecutive displacements rather than a discrete reordering of features. These advances may enable end-to-end training of the model, leading to more data-driven pipeline. Moreover, soft version of alignment instead hard permutation can make this operation differentiable. This transforms the one-on-one linking between adjacent frames into distribution-basee pattern, which is more intuitive.
6 Conclusion
In this paper, we investigate temporal modeling approaches from the perspective of the sufficiency and the efficiency of opatial-temporgg interactions. Through theoretical analysis, we find that increasing mutual information in neighboring frame representations can potentially promote task-relevant information collected by the model, and thus benefits its performance. Then we propose a general alignment and de- de-alignment scheme to enlarge the mutual information among consecutive frames. For specific implementation, we propose our Alignment -uiiedd Temporal Attention (ATA) to conduct feature-level alignment and similarity-based temporal modeling. Extensive experiments show the effectiveness and generality of ATA. Our proposed module not only achieves satisfying results on multiple video action recognition benchmarks, but also can act as a general plug-in to extend any image backbone for video learning tasks.

---

References
[1] R. Arandjelovic, Gronat, A. Torii, T. Pajdla, and Sivic. Netvlad: Can architecture for weakly supervised place recognition. In CVPR, 2016.
[2] A. Arnab, M. Dehghani, G. Heigold, C. Sun, M. Lučić, and Schmid. Vivit: video vision transformer. In ICCV, 2021.
[3] G. Bertasius, H. Wang, and L. Torresani. Is space-time attention all you need for video understanding? In ICML, 2021.
[4] D. Bertsekas. A algorithm for the assignment problem. Math Program, 1981.
[5] Carreira A. Zisserman. Quo vadis, action recognition? new model and the kinetics dataset. In CVPR, 2017.
[6] S. Chen, E. Xie, Ge, D. Liang, and P. Luo. Cyclemlp: mlp-like architecture for dense prediction. ICLR, 2021.
[7] N. Dalal and B. Triggs. Histograms of oriented gradients for human detection. In CVPR, 2005.
[8] N. Dalal, B. Triggs, and C. Schmid. Human detection using oriented histograms of flow and appearance. In ECCV, 2006.
[9] Donahue, Anne Hendricks, S. Guadarrama, M. Rohrbach, S. Venugopalan, Saenko, and T. Darrell. Long-term recurrent convolutional networks for visual recognition and description. In CVPR, 2015.
[10] A. Dosovitskiy, I.. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, Uszkoreit, and N. Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. ICLR, 2021.
[11] H. Fan, B. Xiong. K. Mangalam, Y. Li. Z. Yan, J. Malik, and C. Feichtenhofer. Multiscale vision transformers. ICCV, 2021.
[12] C. Feichtenhofer, H. Fan, J. Malik, and K. He. Slowfast networks for video recognition. In ICCV, 2019.
[13] B. Fernando, E. Gavves, J. Oramas, A. Ghodrati, and T. Tuytelaars. Rank pooling for action recognition. PAMI, 2016.
[14] R. Goyal, S. Ebrahimi Kahou, V. Michalski, J. Materzynska, S. Westphal, H. Kim, Haenel, I. Fruend, P. Yianilos, M. Mueller Mueller-Freitag, et al. The" something something" video database for learning and evaluating visual common sense. In ICCV, 2017.
[15] X. Guo, X. Guo, and Y. Lu. Ssan: Separable self-attention network for video representation learning. In CVPR, 2021.
[16] B. K. Horn and B. G. Schunck. Determining optical flow. Artif Intell, 1981.
[17] E. Ilg. N. Mayer, T. Saikia, M. Keuper, A. Dosovitskiy, and T. Brox. Flownet 2.0: Evolution of optical flow estimation with deep networks. In CVPR, 2017.
[18] A. Jabri, A. Owens, and A. Efros. Space-time correspondence as contrastive random walk. NeurIPS, 2020.
[19] T. Jebara. Kernelizing sorting, permutation, and alignment for minimum volume pca. In COLT, 2004.
[20] S. Jiang D. Campbell, Y. Lu, H. Li, and R. Hartley. Learning to estimate hidden motions with global motion aggregation. In ICCV, 2021.
[21] A. Karpathy, G. Toderici, Shetty, Leung, R. Sukthankar, and Fei-Fei. Large-scale video classification with convolutional neural networks. In CVPR, 2014.
10

---

[22] W. Kay, J. Carreira, K. Simonyan, B. Zhang, C. Hillier, S. Vijayanarasimhan. F. Viola, Green, T. Back, Natsev, al. The kinetics human action video dataset. arXiv preprint arXiv: 1705.06950 2017.
[23] A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. NeurIPS, 2012.
[24] A. Kushwaha and A. Khare. Human activity recognition by utilizing local ternary pattern and histogram of oriented gradients. In ICBMA, 2021.
[25] H. Kwon, M. Kim, S. Kwak, and M. Cho. Motionsqueeze: Neural motion feature learning for video understanding. In ECCV, 2020.
[26] I. Laptev. On space-time interest points. IJC,, 2005.
[27] X. Li, Wang. X. Li. and Y. Lu. Video instance segmentation by instance flow assembly. IEEE Trans Multimedia, 2022.
[28] R. Lin, J. Xiao, and J. Fan. Nextvlad: An efficient neural network aggregate frame-level features for large-scale video classification. ECCV Workshops, 2018.
[29] C. Liu, J. Yuen, and A. Torralba. Sift flow: Dense correspondence across scenes and its applications. PAMI, 2011.
[30] Z. Liu, Lin, Y. Cao, H. Hu, Wei, Zhang, S. Lin, and B. Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, 2021.
[31] Z. Liu, H. Mao, C.-Y. Wu, C. Feichtenhofer, T. Darrell, and S. Xie. A convnet for the 2020s. In CVPR, 2022.
[32] Z. Liu, J. Ning, Y. Cao, Y. Wei, Z. Zhang, S. Lin, and H. Hu, Video swin transformer. In CVPR, 2022.
[33] D. Neimark, O. Bar, M. Zohar, and D. Asselmann. Video transformer network. In ICCV, 2021.
[34] M. Patrick, D. Campbell, Y. M. Asano, I. M. F. Metze, C. Feichtenhofer, A. Vedaldi, and J. F. Henriques. Keeping your eye the ball: Trajectory attention in video transformers. In NeurIPS, 2021.
[35] J. S. Pérez, E. Meinhardt-Llopis and G. Facciolo. Tv-ll optical flow estimation. IPOL, 2013,
[36] D. Ramanan, D. A. Forsyth, and A. Zisserman. Strike pose: Tracking people by finding stylized poses. In CVPR, 2005.
[37] M. S. Ryoo, A. Piergiovanni. A. Arnab, M. Dehghani, and A. Angelova. Tokenlearner: Adaptive space-time tokenization for videos. In NeurIPS, 2021.
[38] X. Shi, Z. Chen, H. Wang. D.-Y. Yeung. W.-K. Wong, and W.-c. Woo. Convolutional Istm
network: A machine learning approach for precipitation nowcasting. NeurlPS, 2015.
[39] D. Sun, S. Roth, and M. J. Black. Secrets of optical flow estimation and their principles. In CVPR, 2010.
[40] D. Sun, X. Yang, M.-Y. Liu, and J. Kautz. Pwe-net: Cnns for optical flow using pyramid, warping, and cost volume. In CVPR, 2018.
[41] Z. Teed and J. Deng. Raft: Recurrent all-pairs field transforms for optical flow. In ECCV, 2020.
[42] D. Tran, L. Bourdev, R. Fergus, L. Torresani, and M. Paluri. Learning spatiotemporal features with 3d convolutional networks. In ICCV, 2015.
[43] H. Wang and Schmid. Action recognition with improved trajectories. In ICCV, 2013.
[44] H. Wang, A. Kläser, C. Schmid, and C.-L. Liu. Dense trajectories and motion boundary descriptors for action recognition. IJCV, 2013.

---

[45] H. Wang, X. Guo, Z.-H. Deng, and Y. Lu. Rethinking minimal sufficient representation in contrastive learning. In CVPR, 2022.
[46] L. Wang, Y. Xiong, Wang, Y. Qiao, D. Lin, X. Tang, and L. V. Gool. Temporal segment networks: Towards good practices for deep action recognition. In ECCV, 2016.
[47] P. Weinzaepfel, J. Revaud, Z. Harchaoui, and C. Schmid. Deepflow: Large displacement optical flow with deep matching. ICCV, 2013.
[48] M. Wu, B. Jiang, D. Luo, J. Yan, Y. Wang, Y. Tai, C. Wang, J. Li, Huang, and X. Yang. Learning comprehensive motion representation for action recognition. AAAI, 2021.
[49] Xie, C. Sun, Huang. Z. Tu, and K. Murphy. Rethinking spatiotemporal feature learning: Speed-accuracy trade-offs in video classification. In ECCV, 2018.
[50] Y. Zhang, X. Li, C. Liu, B. Shuai, Y. Zhu, B. Brattoli, H. Chen, I. Marsic, and J. Tighe. Vidtr: Video transformer without convolutions. In ICCV, 2021.
[51] J. Zhou, Y. Cao, X. Wang, P. Li, and W. Xu. Deep recurrent models with fast-forward connections for neural machine translation. TACL, 2016.
12

---

Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section 5
(c) Did you discuss any potential negative societal impacts of your work? [No]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Section 4.1
(c) Did you report error bars (e.g.. with respect to the random seed after running experi- ments multiple times)? [No]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [No]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [No]
(c) Did you include any new assets either in the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're
using/curating? [No]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [No]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "eqiHNNRMDcb", "reviewer": "Reviewer_1mVh", "summary": "This paper tackles the video action recognition task focusing on temporal modeling. An Alignment-guided Temporal Attention (ATA) module with similarity-based feature-level alignment is proposed to enlarge the mutual information among consecutive frames, and theoretical analysis shows that increasing mutual information in neighboring frames' feature representations can increase task-relevant information and benefit the model performance. Experiments are conducted on two video action recognition datasets, i.e., Kinetics-400 and Something-Something V2, and show that the proposed ATA module can be a general plug-in module to apply on any image backbones for video recognition task.\n\n\n", "questions": "1. For the results in Table 4, why are the results in the last line highlighted? The numbers in that line are not the best compared to others.\n\n2. It is better to mark these symbols from equations (17) and (18) into the Figure 5 for easy understanding.", "limitations": "The authors mention the privacy issue of using these video action recognition models at certain circumstances, e.g. hotels and homes,\nin the part of potential negative societal impacts. While the general solution of restricting the model usage permission is pointed, the specific policies still depend on multiple factors, e.g. social science, etc.\n\n\n", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- Strengths:\nThe theoretical analysis is a plus, showing that increasing mutual information in neighboring frames' feature representations can increase task-relevant information and bring performance improvement.\n\n- Weaknesses:\n1. the proposed ATA module contains one alignment component and one de-alignment component. Though it is motivated in L183 that “To keep the original location-based information and facilitate subsequent spatial operations, a recovery with de-alignment as in Eq. (7) is thus needed.\", how will the de-alignment component affect the performance?\n\n2. The main results shown in Table 1 and Table 2 do not show significant improvement compared to other models, without convincing justification for the relative inferior results. Some explanations like in L214 that\"Our approach can achieve better performance if using pretrained models on larger datasets, e.g., JFT-300M.\"  just show the anticipation without the real result support.\n\n3. For the ablation study of ATA module in different locations shown in Table 3, the ATA module is inserted in an increasing order. How about the independent performance of inserting ATA module into different locations?\n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "N1P8dQfPYjH", "reviewer": "Reviewer_Shks", "summary": "This paper presents a method for video action recognition. It discusses the problems of factorized (2D+1D) and joint (3D) spatial-temporal operations and proposes Alignment-guided Temporal Attention (ATA) for temporal modeling. ATA achieves strong performance while maintaining low computational overheads. What’s more, this paper theoretically proves that the frame-by-frame alignment increases mutual information between neighboring frame representations, thereby potentially including more task-relevant information. Extensive experiments show the effectiveness and generality of ATA, which not only achieves satisfying results on multiple video action recognition benchmarks, but also can act as a general plug-in to extend any image backbone for video learning tasks.", "questions": "1.\tThe idea of aligned temporal attention is reasonable, however, the alignment method seems to be not innovative enough. It seems that the pixel alignment method in this paper is as same as that in reference [4]. Could you please clarify the difference of these two alignments?\n2.\tCould you please explain more about the Figure 1 (a)? How can the Task & representations shared information (illustrated by red line) be obtained? Why is it a trapezoid? Is the Task-relevant information obtained from RGB videos?\n3.\tIt seems that the paper doesn’t cite the SOTA results to make the comparison, for example the articles below should be cited and compared.\n-\tLiu Z, Ning J, Cao Y, et al. Video swin transformer. CVPR. 2022.\n-\tLi K, Wang Y, Gao P, et al. Uniformer: Unified transformer for efficient spatiotemporal representation learning. ICLR, 2022.\n4.\tWhen calculating the alignment with Kuhn-Munkres Algorithm, cosine similarity is employed. Why the cosine similarity is chosen? Could you give some ablation studies based on the choice of similarity?\n", "limitations": "1.\tThis paper indicates that the aligned temporal attention can mine more task-relevant information from the aligned pixels. However, the spatial-temporal self-attention structure can model the attention among all the pixels along the time axis. Consequently, it seems that the alignment doesn’t have much significance.\n2.\tIt seems that the results don’t have a huge improvement. For example, in Table 1, there is no large difference between the results of MViTv1-B, ViViT-L, Mformer-HR and ATA for Top-1 Acc.\n", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nThe paper is well organized and the writing is clear. The figures are well-understood. The experiments seem to be limited but convincing. \nWeaknesses:\nThis paper doesn’t propose a model which is original and innovative enough. Because the main contribution is the aligned temporal attention operation, however the alignment operation seems to be as same as the alignment method in reference [4].\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "6PwBXkVpsBI", "reviewer": "Reviewer_Fgtq", "summary": "This paper introduce a novel alignment-guided transformer architecture for video action recognition. Specifically, the proposed method makes use of KMA to obtain the alignment matrix. Then 1D temporal operation is further used to extract better video representation. The proposed method achieves competitive results on mutiple action recognition datasets.", "questions": "Q1.Have the reviewers investigates how the temporal sampling rate affects the proposed method?There are two ways to examine this.\n1. If the input is temporally downsampled by x2 x4 x8, how does this affect the model performance\n2. the ATA is done on every other frame, what is the performance difference\n\nQ2. Does the model learn better motion representation? This can be done by conducting the arrow of time experiment as in [A,B]\n[A] Xie, Saining, et al. \"Rethinking spatiotemporal feature learning: Speed-accuracy trade-offs in video classification.\" Proceedings of the European conference on computer vision (ECCV). 2018.\n[B] Zhou, Bolei, et al. \"Temporal relational reasoning in videos.\" Proceedings of the European conference on computer vision (ECCV). 2018.\n\nQ3. How does the global and local motion affects the method. According to the authors, the model performs better on ssthv2. Is this because the videos in ssthv2 have more global motion. It would be great to see some analysis on this.", "limitations": "1. This paper needs some additional analysis on what has been learned by the proposed method. See questions raised in Questions section.\n2. This paper should better describe the efficiency part. Though the proposed method is parameter free, how ever it does incur quite some additional computational cost when comparing with Timesformer backbone as shown in Table2. The authors should better tune the tone to help readers understand the additional computational cost", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "+the core idea of using KMA to align spatial information for temporal operation is very interesting and novel to my best knowledge.\n\n+The paper is overall well-written.\n\n+The mathematical derivation is solid, and the results are strong.\n\n+The proposed method is robust and can achieve consistent performance improvement using different backbones.\n\n\n-The analysis on what has been learned by the model is not thorough enough, see questions section\n\n-some details should be presented with more clarity see limitation section\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "2gOZ_O-bMYi", "reviewer": "Reviewer_Ppgh", "summary": "The paper tackles the problem of modeling temporal context from the input frames for video action recognition task. It discusses the current limitations of existing approaches based on factorized and joint convolutions. As a result, it proposes to use frame-by-frame alignment to potentially increase mutual information of frame representations. The proposed Alignment-guided Temporal Attention (ATA) is studied on two widely used action recognition datasets, Kinetics400 and Something-Something V2 showing improvement over baselines models. ", "questions": "1) Why there is an increase in computational complexity over baseline TimeSformer with 8 frames. Is it just using a larger number of input frames (32)?\n2) Would be more beneficial to have a more detailed comparison and discussion with different attention methods like vanilla attention and some factorized and more efficient versions of attention. ", "limitations": "I find the discussion of the limitations is not sufficient and detailed. ", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**:\n1) The paper is well-written with nice and useful visualizations. The motivation for the proposed method is clear. \n2) The discussion of the limitations of the current standard convolutional approaches for temporal modeling is thorough. The theoretical results help to better understand the internal benefits of the method.\n3) Ablation study shows the improvement over simple averaging and another attention-based method. \n4) Combination of ATA with TimrSformer achieves state-of-the-art results, outperforming even larger variants of TimeSformer (TimrSformer-L and TimrSformer-HR).\n\n**Weaknesses**:\n1) I would like to see more discussion and comparison with attention-based methods? How does frame-by-frame alignment help in comparison to global attention? The complexity of the Kuhn-Munkres Algorithm is also cubic as in global attention. So what are the benefits?\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept