TokenLearner: Adaptive Space-Time Tokenization for Videos
Authors: Michael S Ryoo
AJ Piergiovanni
Anurag Arnab
Mostafa Dehghani
Anelia Angelova
Venue: NeurIPS 2021 Poster (2021)
Keywords: visual representation
transformers
video understanding

Abstract: In this paper, we introduce a novel visual representation learning which relies on a handful of adaptively learned tokens, and which is applicable to both image and video understanding tasks. Instead of relying on hand-designed splitting strategies to obtain visual tokens and processing a large number of densely sampled patches for attention, our approach learns to mine important tokens in visual data. This results in efficiently and effectively finding a few important visual tokens and enables modeling of pairwise attention between such tokens, over a longer temporal horizon for videos, or the spatial content in image frames. Our experiments demonstrate strong performance on several challenging benchmarks for video recognition tasks. Importantly, due to our tokens being adaptive, we accomplish competitive results at significantly reduced computational cost. We establish new state-of-the-arts on multiple video datasets, including Kinetics-400, Kinetics-600, Charades, and AViD.

TokenLearner: Adaptive Space-Time Tokenization for Videos
Michael S. Ryoo!!2, AJ Piergiovanni', Anurag Arnab', Mostafa Dehghani', Anelia Angelova
Google Research
Stony Brook University
{mryoo, ajpiergi aarnab, dehghani, dehghani, aangaa))0o..l1111100ogg.......
Abstract
In this paper, we introduce novel visual representation learning which relies on handful of adaptively learned tokens, and which applicable both image and video understanding tasks. Instead of relying on hand-designed splitting strategies to obtain visual tokens and processing a large number of densely sampled patches for attention, our approach learns to mine important tokens in visual data. This results in efficiently and effectively finding few important visual tokens and enables modeling of pairwise attention between such tokens, over a longer temporal horizon for videos, or the spatial content image frames. Our experiments demonstrate strong performance on several challenging benchmarks for video recognition tasks. Importantly, due our tokens being adaptive, we accomplish competitive results at significantly reduced computational cost. We establish new state-of-the-an on multiple video datasets, including Kinetics-400, Kinetics-600, Charades, and AViD.
The code will be available at:
Introduction
Videos provide an abundance of visual information. Vidco understanding particularly requires employing effective apatial-teppone processing of frames capture long-range interactions (5)
37
20 25 []. An important aspect of this understanding is how to quickly learn which parts of the input video stream are important, both spatially and temporally, and to focus computational resources on them. But what basic processing mechanism are able to do so successfully?
21
12
34
Recent advancements in Transformers demonstrate improved accuracy on vision classification tasks. For example, departing from standard convolutional approaches, the Vision Transformer (ViT) [ treats the image as a sequence of patches, utilizing the Transformer architecture (39] similar text understanding. Standard approaches for video recognition take videos as stacked images (i.e., space- time volume) and tend extend 2D neural architectures to 3D (e.g., 3D-ResNets [17 5 38 11).
Motivated by ViT, recent approaches 2 3 also extend Transformers for videos by creating 3D 'tubelet" video tokens with regular 3D-grids, which often result in computationally heavy models. There are often too many tokens to process, especially for longer videos.
The main question addressed in this work is how to adaptively learn the representation from visual inputs to most effectively capture the spatial information for image frames and spatio-temporal interactions for videos. Here are our main ideas:
The first key observation is able learn to represent visual data by learning 'tokenize' the representations. This in contrast to previous approaches which used densely sampled tokens e.g., 16x16 32x32
series of attention layers 9 3
35th Conference on Neural Information Processing Systems (NeurIPS 2021).

---

1xtxC
Learned tokens
Spatial pooliing
Scotia pooling
poolingg
HoWk
HAWK
HalWo
Spatial attention
0.0K)
HeWoC
Input tensor
Figure Visual illustration of the Tokenl earner module, applied to single image frame. kenleerner learns to spatially attend over subset of tensor pixels (i.e., from intermediate spatial representations), and generates a set of token vectors adaptive to the input.
input image/video, making the tokens adapt the input data. We compute multiple spatial weight maps per frame with spatial attention mechanism, and use it for the tokenization. The goal of these maps learn which areas are of importance. Here, each spatial weight map is multiplied with the input to form 'token', to be processed by the subsequent learning modules.
Specifically, we can learn to compute important regions
Furthermore, we find that very few tokens may be sufficient for a visual understanding task. More specifically, show that one significantly reduce the computational budget of video Transform- ers, by utilizing 8-16 tokens as an intermediate frame representation (instead of keeping 200~500). Our TokenLearne is able to reduce the number of total FLOPS by half, while maintaining increasing the classification accuracy.
The approach is simple, efficient, and, shown by the results, outperforms methods including both convolutional methods and previous space-time Transformer ones from prior art. In video under- standing tasks, we establish new state-of-the-art numbers on Kinetics-400, Kinetics-600, Charades, and AVID datasets by outperforming prior models.
2 LerenLe Modules for Adaptive Tokenization
In visual Transformer architectures such ViT (9) an input image is first tokenized by splitting into small (e.g.. 16x 16x16) spatial patches, which are used as input to the model. Similarly, in recent video Transformer architectures, such ViVIT 2 and TimeSformer [3], the video is tokenized by cutting the video into 2d spatial or 3d spatio-temporal cubes on a regular grid.
Instead of processing fixed, tokenized inputs, our attention module learns the tokens that are to be used for the recognition task. We gain several important properties by doing so: (1) We enable the adaptive tokenization so that the tokens can be dynamically selected conditioned on the input. (2) This also effectively reduces the total number of tokens for the transformer, which is particularly beneficial considering that there are many tokens in videos (c.g., 14 x 14 x 64) and the computation quadratic the number tokens. (3) Finally, we provide an ability for each subsequent layer to learn to rely on different space-time tokenizations, potentially allowing different layers to capture different aspects of the video. These dynamically and adaptively generated tokens can be used in standard transformer architectures such as ViT for images and ViViT for videos.
2.1 TokenLearner
Let X be input tensor with space-time shape: X TTTxxxxxC where H W corresponds to the spatial dimension the input, T the temporal dimension (i.c., number frames), and is the number of channels. Let X, be temporal slice of it, corresponding to the frame t: X₁ R x

---

In the case of an image imput, T 1 and X = X5. Note that X could also be an intermediate representation within network, and X, will be its slice in such case.
For every time frame t, we learn to generate a series of tokens, Z, = - from the imput frame X, Specifically, we formulate a tokenizer function, 2; AX,)),, which maps the imput frame X₁ to token vector ## xc + R The idea is to learn our tokenizer function A, to adaptively select an informative combination of pixels (or spatial locations) in X,, and we have S number of such functions. This way, our tokens will not be fixed splits of the input tensor, but set of adaptively changing spatial selections. Different tokens will be mined per frame, allowing us to model their space-time time relations/nnteraciions in case of videos. We also set S to be smaller than H x W (e.g.. and H W 14 14), enabling the model to significantly reduce the computations needed for the layers following this module.
Here, tokenizer 2; (xx)) is implemented with spatial attention mechanism: i.e., the model learns to compute weight map (of size H x W) conditioned on the input Xt, and is multiplied with X, itself. More specifically, let o; (X₂) be function generating the spatial H W x weight map.
Each token 2j is generated by
2, = Ax)), = p(X, O A,w) = p(X, @ ((a,(X)))),
(1)
where @ is the Hadamard product (i.e., element-wise multiplication) and Aiw &RHWXC is an intermediate weight tensor computed with the function as (X,) and the broadcasting function 7-)-
Finally, spatial global average pooling p(-) is applied on top of them to reduce the dimensionality to
R The resulting tokens are gathered to form the output tensor: Ze = [ E SXC
The overall process has a form of an element-wise spatial self-attention. In our version, {a)())) are implemented together followed by a sigmoid function, although this could extended with other implementations. In case of an image, Z = Z.. In the case of a video, the tokens Z, from all the frames are collected to form the final output token tensor E ST
single
series of convolutional layers (with the channel size S)
We specifically name our token learning module as "Tokenl Figure visually summarizes the TokenLeearnel module.
Compute reduction in Transformers: The learned tokens (i.e., the outputs of the TokenI 7) are provided to the subsequent layers the visual representation learning, such as multi-head self- attention (MHSA) used in Vision Transformer and ViViT. With the layers only need to process a small number of tokens (e.g., 8 instead of 1024 per frame) and this significantly reduces the computations, as they are quadratic to the number of tokens. Figure 4 (a) shows basic architecture inserting the TokenLearner module within ViVIT. It could be added at any location within the network, and the relative compute of the Transformer layers after the TermnL become almost negligible due the huge difference in the number of tokens.
Learner, these subsequent
2.2 TokenFuser
After the TokenL generates tokens and its subsequent Transformer layer (e.g.. MHSA) processes them, the "TokenFuser" could be used to further (1) fuse information across the tokens and (2) remap the representation back to its original spatial resolution. This enables the model to capture spatial (or spatio-temporal 'patterns' formulated by the tokens, and recover the original input tensor shape when necessary.
Transformer output: SxC
HWxS
HkWxC
HxWxC
Figure 2: Visual illustration of the TokenFuser module, applied to each image frame individually.

---

Learned Tokens
HoWWxC
Reshape linear
HXWXC
Taken
STxC
Altendion lansor
STxSTXC
Vike
Linear loyer over ST: Rel R
TalennFuse" modure
I+2
Multi-head Attention (or Vector Transformer)
Takee
1-T
TokenLearner
TokenFuser
Figure 3: TokenLearner, Transformer, and TokenFuser combined for video representation learning. TerenLearner first learns to generate a set of token vectors, Transformer (e.g., MHSA) models their space-time relations, and TokenFuser combines them. S is the number of tokens we learn per frame, and 7' is the number of frames. Note that this combination can serve as 'module" itself, and one may stack such module multiple times within the network. TokenFuser could be dropped.
First, given the token tensor Y E ST'C from Transformer layer, apply linear layer (i.e.. a fully connected MLP layer) over the tokens, not channels. That is, we learn a linear function of RST + 12 where S is the number of our tokens mined per frame and 7 is temporal size of the input tensor, and apply it to every channel independently. That is, we update = (Y" M)" where M is learnable weight matrix with size ST x ST. The result of such operation maintains the tensor size ST C. believe this also has a connection to the observations from the concurrent work, MLPMixer 36), that token-wise linear layers are beneficial.
Next, the TokenFuser processes each temporal slice Y, E RSCC individually, and remaps the token tensor of size S x C' back to x W x C, by learning to combine the tokens for each spatial location in H W differently.
X B(Y, x/) B, X/ = 3,(xX)) + XX
where X is the residual input to the previous TokenLearner module, Y, is the processed tokens in the TokenFuser module, and X is the output. B,, E "2'W'x is an intermediate weight tensor computed with the function (X,). The function (X.) is implemented with a simple linear layer followed by sigmoid function.
Figure illustrates the overall process of the TokenFuser (the token-wise linear layer omitted).
2.3 Video architecture overview
Here, we provide an overview of video representation architecture with TokenLearner. The To- kenLearner and TokenFuser modules introduced in Section 2 are directly applicable for video representation learning. TokenLearner generates multiple for frames videos and they are stacked to form Z. Once Z is generated, any standard Transformer layers could be used to parse them jointly.
Figure provides an overview of the combined architecture for video representation, which is to be repeated over multiple layers. TokenLearner first extracts S number of tokens per frame, resulting in a total of ST tokens where T is the number of frames. Once TokenLearner generates these adaptively learned tokens, they provided to the subsequent Transformer layer to capture the global space-time patterns. Finally (and optionally depending on the architecture), TokenFuser applies a linear layer the token axis and remaps the tensor shape back, as discussed in Subsection 2.2 Following Eq. TokenFuser is applied for per-frame representation Y,. This results a lightweight approach, which brings forth efficient vidco representation capturing long-range visual patterns.

---

Experiments: TekenLearner with Video Vision Transformer
3.1 Network architecture implementation
Action class
Action class
In this experiment, former (VIVIT) architecture [2]. following its detailed settings and implementation 77. VIVIT is direct extension of VIT [9] for videos, which uses spatio- temporal tubelets from videos as its tokens. The size the space-time tubelets are typically 16x16x2, which are given to the Transformer layers.
the Video Vision Trans-
Classification head
Classification head
tokons
196'T
Transformer
TokenFuser
tokens
Nespos
Transformer
Transformer
We use ViViT-L/16 as our backbone, while also apply- ing the TokenLearner to backbones with more initial tokens such L/14 and L/10. ViViT-L models have 24 transformer layers, Following the setting of [2), we used the input resolution of 224x224, extracting tubelets, and attaching positional encodings.
tokons
TokenLearner
TokenLearner
Transformer
Transformer
Stem
Stem
Figure (a) and (b) show two different architectures incorporating Terner (a) is formed by in- serting TekenLearner in the middle of the network such as after the 12th layer among 24, while (b) uses both Tokenl Leanner and TokenFuser. In particular, our model (b) is formed by replacing conventional Transformer layers with series of looknnleearne Transformer-TookenFueee Similar to (a), such replace- ment is done only for the layers after certain point. For instance, we keep twelve of the standard Transformer MHSA layers the beginning, and replaces the remaining twelve layers with our TokenLearnent wew.eeeeTTaaarforrmetee Transformer- Teansformer-TTooeFuuuse modules repeated twelve times. We also modified L/14 and L/10 models have more transformer layers (e.g.. 35 instead of 24). Note that the computation increase caused by the transformer layers added after TokenLeamee module is relatively very small, as the number of tokens are few: 8 or 16 per frame.
Video
Video
(a)
(b)
Figure 4: Our models following the ViVIT architecture. (a) with TokenLearner and (b)
with both TokenLearner and TokenFuser.
We tried various number of tokens including = 8, 16, 32, and use S = 8 and 16 as our default settings. That is, the TokenLearner is learning to abstract an image frame into 8 (or 16) tokens. The spatial attention function (a: in TokenLearner is implemented with four 3x3 conv. layers (with gelu in between), whose channel size is identical to the number of tokens (e.g.. S 8).
3.2 Datasets and training
We use the Kinetics datasets, which are video classification datasets with relatively short video clips (~10 seconds). We train and evaluate on both Kinetics-400 and Kinetics-600 datasets, which have about 240k and 390k training samples. We follow the standard settings used in previous papers and report accuracy on the validation set [2].
Following VIVIT 2 we first pretrain models on JFT 355 to obtain initial weights. The weights of the initial convolutional layers handle image patches (e.g.. 16x16) are processed to handle 16x 16x2 video patches by following ViVIT's 3D initialization strategy, and the weights of the Transformer and the TokenLeamee layers are directly inherited.
3.3 Results
We evaluate various versions of the ViT-L models incorporating the TokenLearner module. As mentioned above, all of the models are pre-trained JFT and finetuned on Kinetics. We use the standard L/16 models + carner,, as well as L/14 and L/10. L/14 and L/10 use 11 additional layers compared to the standard ViT L/16, but also described in the above subsections, the computation increase caused by them are minimal due to the number of tokens being much smaller, 8 16 per frame, in the added layers. We report both their classification accuracies and FLOPS.
Table Table compares the accuracies of the base ViVIT models against our VIVIT + TokenLearner models on Kineties-400. These models are directly comparable as they follow the exact same setting and
5

---

Table Comparison ViVIT models with and without TereeL on Kinetics-400, GLOPS are
view. The difference in the number of parameters between the TokenLearner models comes from the different number of layers used after the TorenLearner module.
Method
Top-1 accuracy
Top-5 accuracy
# params.
GFLOPS
ViViT-L/16 [
82.8
95.5
308M
1446
ViViT-L/16 320
83.5
95.5
308M
3992
ViViT-H/I4 22
84.8
95.8
654M
3981
ViViT-L/16 (our run)
83.4
95.6
308M
1446
TokenLeamer 16at12 L/16
83.5
95.6
308M
766
TokenLeamnr 8at18 + L/16
84.5
96.1
383M
1105
TokenLeamner 16at18+ L/14
84.7
96.1
447M
1621
TokenLearner 16at18+ 8+ L/10
85,4
96.3
450M
4076
Table ViVIT TekenLe on Kinetics-400, compared to the state-of-the-art models. Different approaches rely on different pre-training datasets, such as ImageNet-21K (for TimeSformer and
Swin) and JFT (for ViVIT and TokenLearner), The multiplication in GFLOPS correponds to the number of views used for the inference, such as 4x3 = 12.
Method
Top-1 accuracy
Total GFLOPS
R(2+1)D 38
73.9
304 x 115
SlowFast 16x8, R101+NL 12
79.8
234 x 30
TimeSformer-L.
80.7
2380 x
ViViT-L/16
82.8
1446 12
ViViT-H/14
84.8
3981 x 12
Swin-I. 23
83.1
604 x 12
Swin-L. (384)
23
84.6
2107 12
Swin-L. (384)
23
84.9
2107 x 50
TokenLearner 16at12 (L/16)
82.1
766 x 6
TokenLeamer 8at18 (L/16)
83.2
1105
TokenLearner 16at12 (L/16)
83.5
766 x 12
TokenLeamer 8at18 (L/16)
84.5
1105
TokenLearner 16at18 (L/14)
84.7
1621
TekenLeamer 16at18 (L/10)
85,4
4076 x 12
the pre-train dataset. "Token 16at12" means that we have the TokenLearner layer learning 16 tokens, after the 12th Transformer layer. We are able to observe that the use of TokenLeamer enables better classification while also reducing the compute. In particular, inserting TerneLe in the middle of the network (at 12) achieves better accuracy than the base mode, while cutting the computation by (almost) half. addition, having the TokenLearner at the later layer (at 18) achieves even superior accuracy while still performing faster, thanks to its adaptiveness.
Table 2 compares the TokenI eamnrr accuracy against the state-of-the-arts models. Note that these approaches follow slightly different settings and pretrain datasets (e.g., the use of ImageNet-21 K instead of JFT like ours). We believe the accuracy of 85.4 is the highest that has been reported far, and we believe it is meaningful. on Kinetics-400, while also being computationally efficient.
the results on Kinetics-600, Similar to our results able to observe that proposed approach extends the sate-of-thhe-art
Experiments: TorenL with Bottleneck Transformer
4.1 Network architecture implementation
In this experiment, we follow the Bottleneck Transformer 33 network style. while taking advantage of X3D [TT] the backbone. This is motivated by the successful usage of X3D on Charades.

---

Table VIVIT Token on Kinetics-600. The multiplication in GFLOPS correponds the
number of views used for the inference, such 4x3 = 12.
Method
Top-1
Total GFLOPS
SlowFast 16x8, R101+NL 12
81.8
234 x 30
X3D-XL [|
81.9
48 x 30
TimeSformer-Hi
82.4
1703 x
ViViT-L/16 2
84.3
1446 X 12
ViViT-H/14
85.8
3981 X 12
Swin-B 23
84.0
282 x 12
Swin-L (384)
23
85.9
2107 12
Swin-L (384)
86.1
2107 x 50
TokenLcarner 16at12 (L/16)
84.4
766 X 12
TokenLearner 8at18 (L/16)
86.0
1105
12
TokenLearner 16at18 (L/10)
86.1
4076 12
TokenLearner 16at18 w. Fuser (L/10)
86.3
4100 X 12
128-d out
Specifically, modified X3D to be more computationally efficient by (1)
replacing its 3D XYT convolutional layers with pair of 2D conv. layer and ID conv. layer, and (2) removing gquceze-and-Exxitation layers (18] and swish activations. Our backbone could be viewed as X(2+1)D, We the channel sizes and the number of layers identical to X3D-M, which an efficient model.
492, 1x1x1, 128
49"T tokens
TokenFuser
Based such 1(2+1)D architecture, and following the Bottleneck Trans- former concept, we replace the space-time convolution layers in the last block with our transformers. Figure 5 illlstrrates the residual module architecture, which repeated multiple times in the block. Terenr, Transformer, TokenFuser are applied in sequence, with optional 2D 3 x convolution layer before them. The spatial attention function (i.e., a-)) in TokenLeamer is implemented with single conv2d layer.
Vector Transformer
tokens
TokenLeamer
Here, used Vector Transformer instead of MHSA layer, which could be also viewed as the MHSA with the number of heads being identical to the number of channels. We provide more details in Appendix.
Transformer
(optional) Conv2D
tokens
128, 1x1x1, 492
We use 224 x 224 x 64 videos for training and 256 x 256 64 videos for testing. After the 3rd residual block. the input tensor has the shape of 8 x 64, and this becomes the input to the TokenLearner. For an efficient implementation the intermediate channel size of TokenLearner was identical to the output channel size, d = 432. Notice that 64 frames were used best capture longer-term temporal information. S number of tokens were used.
128-d in
Figure 5: Our network module following the bot- tleneck transformer, with X(2+1)D backbone. It is an inverted bottleneck.
4.1.1 Datasets
The Charades dataset [3 is dataset collected by assigning activity tasks which people in various environments acting out, by performing sequence of actions which involve interaction with objects. For example, sitting on the couch and reading a book, closing the book, standing up and speaking the phone. It comprises 8000 training and 1686 validation videos with an average duration 30 seconds. has 157 activity classes. This dataset very challenging as is multi-class, multi-label video dataset, that is, more than one activity occur at the same time, and it includes fine grained motions or interactions with small objects in real-world environments. We follow the standard evaluation protocols, reporting the mean Average Precision (mAP) % (v1 classification setting of the dataset). We used the frame rate of fps and 12 fps to obtain the training/testing videos. The dataset has Con-Cmmmrrlial Use license.
Charades dataset:
7

---

Table Performance on the Charades multi-labed classification task. 12 fps setting. Performance is measured using the Mean Average Precision (mAP) since more than one ground truth action is possible. Methods with RGB and optical flow input modalities are listed.
Method
Input
Pre-train
mAP
I3D [3]
RGB
Kinetics
32.9
I3D from 40
RGB
Kinetics
35.5
I3D + Non-local 40
RGB
Kineties
37.5
EvaNet [26
RGB
Kinetics
38.1
STRG 41
RGB
Kineties
39.7
LFB-101 43
RGB
Kinetics
42.5
SGFB-101 19
RGB
Kineties
44.3
SlowFast-101 [12
RGB+RGB
Kinetics
45.2
AssembleNet-50 30
RGB+Flow
None
47.0
Multiscale ViT [10
RGB
Kinetics
47.7
AssembleNet-101 330
RGB+Flow
Kineties
58.6
AssembleNet++ 29
(w/o object)
RGB+Flow
None
55.0
MoViNets 22
RGB
None
63.2
Backbone (X(2+1)D-M)
RGB
None
62.7
Ours
RGB
None
66.3
Table 5: Performance on the Anonymized Videos from Diverse countries (AViD) dataset. Performance in terms of mean accuracy is shown in % averaged over 887 classes. Previous approaches results are reported from [27]. all based on training from scratch with RGB-only inputs.
Method
Accuracy
total GFLOPS
I3D [5]
46.5
108 N/A
(2+1)D ResNet-50
46.7
152 x 115
3D ResNet-50
47.9
N/A
SlowFast-50 8x8 [12
50.2
65.7 x 30
SlowFast-101 101 16x4
212
50.8
213 30
Backbone (X(2+I)D-M)
48.6
532
X(2+1)D-M w/ disjoint space+time Transformer (like
50.6
493 x
Ours
53.8
487
AViD dataset: The Anonymized Videos from Diverse countries (AViD) dataset 27 is unique dataset which is representative of the world's population video content generation. from videos uploaded from multiple countries across six continents and demonstrates higher diversity compared to other video datasets such as Kinetics in its concepts, actions and visual representations. For example 'greeting" in certain countries involves handshake, in some kiss, but in others a slight bow. The dataset is explicitly designed to contain less bias, encourage diversity, while respecting privacy and licenses. The AViD dataset contains 887 classes and 450k videos (410k training 40k testing) and is of comparable size to Kinetics-400 and Kinetics-600 datasets with 400 and 600 classes respectively, also containing variable duration videos 3 - 15s. We report classification accuracy over 887 classes, All videos this dataset have the Creative Commons License.
collected
4.2 Results
In Table 4 we compare the proposed TokenLearner to the state-of-the-art methods. Our approach outperforms these, including several recent works. The mAP of 66.3% on Charades classification establishes the new state-of-the-art.
Charades dataset results:
AViD results: Table shows the results the AViD dataset. As seen, approach outperforms prior work on this challenging dataset too. We also compared ours to the reimplementation of TimeSformer module [3] applied to the same backbone ours. This uses disjoint spatial and temporal transformer modules, which was also tested in [2]. We are able to observe that we establish the new state-of-He state-of-the-art the-arts on this dataset, while also being more computationally efficient.

---

Table Comparison between TekenLearner and the joint space-timi time transformer modules similar to 2]. applied to our backbone. They use the X(D+I)D backbone, tested on Charades with the 6 fps setting, Charades 12 fps setting, and AViD dataset. GFLOPs and # params are of each module (with 64 frame inputs), not the entire network.
Module
Char-6fps
Char-12fp:
AViD
GFLOPs
# params
Joint space-time time MHSA Conv2D + Joint space-time MHSA
57.9
64.0
53.3
22.0
0.30M
58.6
62.5
52.5
35.8
1.98M
Ours (TokenLearner)
58.8
63.4
53.8
3.4
0.81M
Ours (Conv2D + TokenLearner)
59.6
66.3
53.7
17.2
2.49M
4.3 Ablations
Comparison against different tokenizations: Here, compare the model with TokenLeamer against space-time transformer modules with the standard tokenization. More specifically, we compare the use of TokenLearner + Vector Transformer + TokenFuser against the full joint space-time transformer module (advocated in [2] and also mentioned in [3), without token learning. The full joint space-time transformer module is transformer layer on space-time tokens similar to ours, but relies only on the hand-designed tokenization. Compared to TokenLearner which generates S x T number of tokens, the full joint space-time transformer uses H x W x T number of tokens. In our bottleneck implementation, it uses ~-8 times more tokens (i.e., 8464 vs. 8*8*64), For the joint space-time transformer modules, the standard multi-head self-attention (MHSA) with 8 heads is used.
Table shows the results. Interestingly, despite the heavier computation of the full joint space- time transformer, it performed slightly worse to the TokenI leanerr modules. We believe this shows the advantage of the 'adaptiveness" of the tokens in the Tokenl Learnrr and shows that the standard transformers might be suffering from the tokens irrelevant to the actions serving as noise or distractors.
We also report the amount of computation and the number of parameters of each module in these models. This depends on the input size and the hyper parameter setting, and our measurement is based on the input size (i.e., T x H x W x C) of 8 x 8 x 64 x 492. Note that this is the measurement of modules, not the entire network.
Comparison between multiple space-time layer combinations. As also suggested in previous literature, a common strategy for video representations to pair a layer focusing on spatial information with layer focusing on temporal information (e.g., R(2+1)D B8] and TimeSformer (3). Table [ shows the results of this ablation. For spatial and temporal transformer implementations, the standard multi head self-attention proposed TokenLeamn is more accurate than other popular combinations. The modules based on TokenLearner also effectively only uses fraction of the Tokens per frame (i.e., 8) as opposed to other methods which use 16 x 16 or 32 x 32 tokens.
used, was done in 2 3 The result shows that the
One of the main benefits of the TokenLearner Learner (in addition to the adaptive tokenization of the input and that we explicitly fuse the tokens to capture their spatio-temporal patterns) that, unlike the disjoint space/time transformers used in this ablation study, it is joint space-time transformer. Simultaneously, still manages its computation to be much more tractable (as shown in Tables and 7: A naive full version of the space-ting prace-time transformer would require consideration of 8 x x 64 = 4096 tokens in our case, building and multiply the attention tensor of size 4096 4096. On the other hand, the TokenLearner learns to consider 8 x 64 = 512 tokens jointly.
More TokenLearner alternatives. We also compared our spatial attention- based token learning with alternative approaches: (1) using fixed grid split each frame into the same number of tokens (i.e.,& 8 tokens), (2) the approach of directly generating tokens using a fully connected layer, and (3) the upproach of spatially average pooling the entire frame pixels and using fully connected layers to generate multiple tokens per frame. In the second approach, we directly model 2, = Ax(x) as a dense layer, producing T x S > C' tensor based on the T x H x W x C' input. The third approach is similar, except that we apply spatial global average pooling per frame and then use MLP to generate tokens.

---

Table 7: Comparison between different space-time transformer modules. They were all applied the same backbone architecture (i.e., the Bottleneck Transformer-sty with X(2+1)D). The tharades-ffpp used in this experiment. FLOPS are estimated with arame settings, per module.
Module
Charades-6fps (%)
GFLOPs
# params
Conv2D + CDnviD
56.6
18.3
2.24M
Conv2D + MLPMixer 36
57.0
13.8
2.06M
Conv2D + Temporal transformer
58,4
16.5
1.98M
Spatial Temporal transformer
58.8
5.5
0.59M
Conv2D + Spatial Temporal transformer
58,0
19.2
2.27M
Ours (TokenLearner)
58.8
3.4
0,81M
Ours (SpatialT TokenLearner)
58.9
6.2
1.11M
Ours (Conv2D + TokenLearner)
59.6
17.2
2.49M
The fixed split tokenization method (1) provided us the accuracy of 58.8 on Charades, as opposed to 59.6 of ours. The direct token generation method (2) provided the accuracy of 56.6 Charades, failing to obtain better tokens. Pooling and generation method (3) gave us the accuracy of 58.6. These results suggest the importance of spatial attention for the token learning, TokenI The same vector transformer and TokenFuser (from Section 2 were used for this ablation.
Related work
Video understanding relies on both the spatial and the temporal information in the video. In or- der to adequately capture both motion and appearance information in videos, full 3D space-time
convolutional layers as well (2+1)DD convolutional layers have been used (37 5 38 44). More
advanced network designs have also been extremely popular in video CNNs particularly two-stream
ones 33
1
14
15
g 12) and, recently, architecture searched ones
30 26).
Attention-based architectures, e.g., the Transformer [3] have shown remarkable success in both Natural Language processing (NLP) and computer vision. Most adaptations of the Transformer architectures to computer vision, have been slow, although some optimizations, have been successful and for video generation 42
e.g., for image classification, 4 45
28
Applying attention-based architectures video presents definite challenge the model needs to learn dependencies across both the spatial and temporal domains. The Vision Transformer [9] demonstrated how the NI.P-specific Transformer architecture can elegantly work for images, by subdividing the input image into non-overlapping patches on regular grid and feeding them as token embeddings to the Trasnformer, where O(N2) tokens are used order of 256 1024. (16) relied on the region proposal network to use the detected human and object candidates as tokens, showing that it could be combined with CNNs.
couple of recent work 3 in the spirit of the Vision Transformer, subdivided the video into token in 3D grid to capture the video input. This leads to O(N3) increase in the number of tokens required for learning (typically ~ 25k tokens for 96-frame model). Our work, in contrast, learns tokens from data which results in a significantly fewer tokens, and more efficient approach. We see that even 8x times fewer tokens (e.g., 512 vs 4096), when learned, are able to capture successfully the information needed for video representation learning.
Conclusions
We have presented TokenLeerner, a novel approach for visual representation learning, which adap- tively tokenizes the representations. The goal is to learn to extract important tokens in image frames and videos for the recognition tasks at hand. Our approach is more efficient, than contemporary work, by finding few important space-time tokens which and videos. We observe improved accuracies across challenging video understanding tasks, and outperformed prior approaches in many datasets. One of the remaining challenges is in learning full spatio-tempore tokens. The current TokenLearner focuses on finding spatial tokens over a sequence of frames, and it could be extended to directly mine tokens over space-time ppace-time volumes.
model visual representations of images
10

---

Acknowledgement
We thank Dmitry Kalashnikov, Andy Zeng, and Robotics at Google NYC team members for valuable discussions attention mechanisms.
References
[1] J.-B. Alayrac, A. Recasens. R. Schneider. R. Arandjelović, Ramapuram, J. D. Fauw, L. Smaira, S. Diele- man, and A. Zisserman. Self-supervised multimodal versatile networks. In Advances in Neural Information Processing Systems (NeurlPS), 2020.
[2] A. Amab, M. Dehghani, G. Heigold, C. Sun, M. Lučié, and C. Schmid. ViViT: A video vision transformer. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2021.
[3] G. Bertasius, H. Wang, and L. Torresani. space-time attention all you need for video understanding? In International Conference on Machine Learning (ICML), 2021.
[4] N. Carion, Massa, Synnaeve. Usunier, A. Kirillov, and Zagoruyko. End-to-end object detection with transformers, In Proceedings of Europaan Conference on Computer Vision (ECCV), 2020.
[5] Carreira and Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
[6] J. Cordonnier, A. Loukas, and M. Jaggi. On the relationship between self-attention and convolutional layers. In International Conference on Learning Representations (ICLR), 2020.
[7] M. Dehghani, A. Gritsenko, A. Arna,, M. Minderer, and Y. Tay. Scenie: A JAX library for computer 2021.
vision research and beyond. arXiv preprint arXiv:2110.
[8] A. Diba, M. Fayyaz, V. Sharma, M. Paluri, J. Gall, R. Stiefelhagen. and L. V. Gool. Holistic large scale video understanding. arXiv preprint arXiv: /904. /455/ 2019.
19] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenbom, X. Zhai, T. Unterthiner, M. Dehghani, M. Min- derer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.
2020.
[10] H. Fan, B. Xiong. K. Mangalam, Y. Li, Z. Yan, J. Malik, and C. Feichtenhofer. Multiscale vision transformers. arXiv preprint arXiv:2104. 11227, 2021.
[11] C. Feichtenhofer. X3D: expanding architectures for efficient video recognition. In Proceedings of the IEFE Conference on Computer Vision and Pottern Recognition (CVPR), June 2020.
[12] C. Feichtenhofer, H. Fan, J. Malik, and K. He. Slowfast networks for video recognition. In Proceedings of the IEEE International Conference Computer Vision (ICCV). 2019.
[13] C. Feichtenhofer, A. Pinz, and R. Wildes. Spatiotemporal residual networks for video action recognition.
In Advances in Neural Information Processing Systems (NeurIPS), pages 3468 3476, 2016.
[14] C. Feichtenhofer, A. Pinz, and R. P. Wildes. Spatiotemporal multiplier networks for video action recognition.
In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4768- 4777, 2017.
[15] C. Feichtenhofer, A. Pinz, and A. Zisserman, Convolutional two-stream network fusion for video action recognition. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), pages 1933-1941, 2016.
[16] R. Girdhar, J. Carreira, C. Doersch, and A. Zisserman. Vidco action transformer network. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2019.
[17] K. Hara, H. Kataoka, and Y. Satoh. Learning spatio-temporal features with 3d residual networks for action recognition. In Proceedings of the ICCV Workshop Action, Gesture, and Emotion Recognition, 2017.
[18] J. Hu. L. Shen, S. Albanie, G. Sun, and E. Wu. Squeeze-ang mnieezze-and-exciaatioo networks. In Proceedings of the IEEE Conference on Computer Vision and Patiern Recognition (CVPR), 2018.
[19] J. Ji, R. Krishna, L. Fei-Fei, and J. C. Niebles. Action genome: Actions as composition of spatio-temporal scene graphs. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
[20] S. Ji, W. Xu, M. Yang, and K. Yu. 3d convolutional neural networks for human action recognition. IEEE Transactions Pattern Analysis and Machine Intelligence, 35(1):221-25 22111211,111111 2013.
[21] W. Kay, J. Carreira, K. Simonyan, B. Zhang, C. Hillier, S. Vjjaanarasimmhan, F. Viola, T. Green, T. Back,
P. Natsev, et al. The kinetics human action video dataset. arXiv preprint wxXvv:1005.06950 2017.
11

---

[22] D. Kondratyuk, L. Yuan, Y. Li, L. Zhang. M. Tan, M. Brown, and B. Gong. Movinets: Mobile video networks for efficient video recognition. In Proceedings the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
[23] Z. Liu, J. Ning. Y. Cao. Y. Wei, Z. Zhang, S. Lin, and H. Hu. Video swin transformer. arXiv preprint arXiv:2106.133 13230. 2021.
[24] M. Monfort, A. Andonian, B. Zhou, K. Ramakrishnan, S. A. Bargal, T. Yan, L. Brown, Q. Fan, D. Gutfruend,
C. Vondrick, al. Moments in time dataset: one million videos for event understanding. arXiv preprint
arXiv: I0w,1.030330 2018.
[25] A. Piergiovanni. A. Angelova, and M. S. Ryoo. Evolving losses for unsupervised video representation learning. In Proceedings of the IEFE Conference Computer Vision and Pattern Recognition (CVPR), 2020.
[26] A. Piergiovanni, A. Angelova, A. Toshev, and M. S. Ryoo. Evolving space-time neural architectures for videos. Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2019.
[27] A. Piergiovanni and M. S. Ryoo. AViD dataset: Anonymized videos from diverse countries. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
[28] P. Ramachandran, N. Parmar, A. Vaswani, I. Bello, A. Levskaya, and J. Shlens. Stand- -land-alone self attf-antention in vision models. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
[29] M. S. Ryoo, A. Piergiovanni, Kangaspunta, and A. Angelova. AssembleNett+: Assembling modality representations via attention conncctions. In Proceedings of European Conference on Computer Vision (ECCV), 2020.
[30] M. S. Ryoo, A. Piergiovanni, M. Tan, and A. Angelova. AssembleNet: Scarching for multi- stream neural connectivity in video architectures. In International Conference on Learning Representations (ICLR), 2020.
[31] G. A. Sigurdsson, G. Varol, X. Wang, A. Farhadi, I. Laptev, and A. Gupta. Hollywood in homes: Crowdsourcing data collection for activity understanding. In Proceedings of European Conference on Computer Vision (ECCV), 2016.
[32] K. Simonyan and A. Zisserman. Two-stream convolutional networks for action recognition in videos. In Advances in Neural Information Processing Systems (NeurlPS), pages 568-576, 2014.
[33] A. Srinivas, T. Lin, N. Parmar, J. Shlens, P. Abbeel, and A. Vaswani. Bottleneck transformers for visual recognition. arXiv preprint urXiv:2101.1 100. 2021.
[34] C. Stroud, D. A. Ross, C. Sun, J. Deng, and R. Sukthankar. D3D: Distilled 3d networks for video action recognition. arXiv preprint arXiv: 18/ 2.088499. 2018.
[35] C. Sun, A. Shrivastava, S. Singh, and A. Gupta. Revisiting unrcasonable effectiveness of data in deep learning era. In Proceedings of the IEEE International Conference Computer Vision (ICCV), 2017.
[36] Tolstikhin, Houlsby, A. Kolesnikov, L. Beyer, X. Zhai, Unterthiner, Yung, A. Steiner. D. Keysers, J. Uszkoreit, Lucic, and A. Dosovitskiy. Mip-mixer: An all-mlp architecture for vision. arXiv preprint arXiv:2 105.01601, 2021.
[37] D. Tran, L. D. Bourdev, R. Fergus, L. Torresani, and M. Paluri, C3d: generic features for video analysis. CoRR, abs/14)2.0767, 2(7)88 2014.
[38] D. Tran, H. Wang. L. Torresani, J. Ray. Y. LeCun, and M. Paluri. A closer look at spatiotemporal convolutions for action recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 6450-6459, 2018.
[39] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention all you need. In Advances in Neural Information Processing Systems (NeurIPS), 2017.
[40] X. Wang, R. Girshick, A. Gupta, and K. He. Non-local neural networks. In Proceedings of the IEFF Conference on Computer Vision and Pattern Recognition (CVPR), pages 7794-7803, 2018.
[41] X. Wang and A. Gupta. Videos as space-time time region graphs. In Proceedings of European Conference on Computer Vision (ECCV), pages 399 417, 2018
[42] D. Weissenborn, O. Täckström, and J. Uszkoreit. Scaling autoregressive video models. In International Conference on Learning Representations (ICLR), 2020.
[43] C.-Y. Wu, Feichtenhofer, H. Fan, He, Krähenbühl, and R. Girshick. Long-term feature banks for detailed video understanding. arXiv preprint arXiv: 2018.
[44] S. Xie, C. Sun, J. Huang, Z. Tu, and K. Murphy. Rethinking spatiotemporal feature learning: Speed- accuracy trade-off's in video classification. In InnPPoocee Proceedings of European Conference Computer Vision (ECCV), pages 305-321, 2018
[45] H. Zhao, J. Jia, and V. Koltun. Exploring self attention for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
12


Reviews:
[{"review_id": "sTW566vIapA", "reviewer": "Reviewer_TXug", "summary": "The paper proposes a transformer-based neural network architecture for extracting video representations. Past approaches discretize entire videos into 3D (RGB-T) chunks as input 'tokens' to transformers. Instead in this work, an additional operation is proposed to extract a smaller number of tokens per frame. This leads to a more computationally efficient architecture that outperforms existing methods on the task of video classification. ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "# Strengths\n\n- The proposed idea of extracting limited number of tokens per frame is intuitive and easy to follow. \n\n- The attention mechanism proposed to extract fixed number of tokens is novel. Due to its simplicity it could be easily adopted in different computer vision applications. Considering the fact that transformers bring heavy computational cost in numerous domains, this idea could potentially be impactful. \n\n- Experimental results demonstrate that the model outperforms existing architectures for video classification on a wide range of multiple datasets. The ablative studies also show that the individual components used, VecT, TokenLearner and the Fusion operator, all contribute towards improve performance of the model. \n\n# Concerns\n\n## Writing\n- The abstract and introduction are not easy to follow. For example, the abstract begins by talking about space-time \"tokens\". To the best of my knowledge, this is not a term used commonly in computer vision. \n\"The proposed approach is designed with the intention to allow the tokenizer to adaptively react to input video frames containing diverse visual content, and then to have the vector transformer and subsequent modules learn the underlying spatio-temporal interactions and long-range dependencies in video inputs.\" \nThis sentence is extremely difficult to follow without any context provided. \n- Line 37: \"Further, with these adaptively learned tokens, we apply a vector attention between key-query pairs.\" So far keys and queries have not been introduced and it seems like the reader is expected to know what \"vector attention\" is.  \n- Line 38: \"This can be thought as a version of Transformer in which the number of heads is the same as channels, allowing us to learn a different attention matrix for each channel.\" This also doesn't make sense without any additional information.\n\n## Approach Details\n- Line 91, why is $|Z| = S$ and not $ST$?\n- Line 99, \"the attention tensor is multiplied with the value matrix\". Here the attention tensor is $ST \\times ST \\times C$ and the value matrix is $ST \\times C$. So what kind of multiplication are you performing here? \n- Line 122-130, since your model is built on the X3D architecture, it would be useful to explain it in more detail here. For readers who are not well-versed with this literature, the rest of the description doesn't make sense. \n\n## Efficiency and Performance\n- Comparing Row 4 and Row 7 of Table 3, it looks like your model is significantly more inefficient while only providing a ~1% gain in performance. In this light, what is the significance of the proposed architecture design choices?\n- Line 202-Line 206, it seems like removing the learned tokenizer and replacing it with a predefined discrete grid only causes a drop in performance of ~0.8%. Again, here it doesn't seem like the TokenLearner is contributing much compared to this naive method. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, the authors have addressed limitations and potential impact thoroughly. \n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3 hours "}, "extra_text": {}}, {"review_id": "Ok2g7iS63q", "reviewer": "Reviewer_h2mC", "summary": "In this paper, the authors present an adaptive space-time tokenization for video representation learning. In the proposed method, each frame in a video is tokenized into a set of visual tokens through weighted summation over spatial locations. Then, vector transformer operates on a collection of tokens from entire video frames to temporally aggregate the information. The updated tokens are added back to the input tensor after linear pooling and broadcasting. The authors validated the proposed network module (TokenLearner) on two video action recognition benchmarks.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n- The idea of tokenizing frames instead of splitting into regular grids looks interesting and working well. The motivation is clear and the proposed approach is potentially beneficial to efficiency and performance of video recognition. \n- The paper contains comprehensive ablation studies that compares the proposed approach with possible alternatives. Visualization of tokenization is also provided.\n- In general, the paper is written clearly and easy to follow.\n\nWeakness:\n- I have a little concern about the technical novelty. Learning an adaptive tokenization of an image is not new. In [A], a very similar mechanism for tokenizing an image into a set of visual tokens are proposed. While this work is proposed for image representation learning, extending it to temporal domain is straightforward. The difference is, in the previous work, they project back the information in the tokens after applying transformer. Projecting the information into the right location looks better than the global update by broadcasting. The authors should discuss this paper.\n- The experiments are weak. The authors use Charades and AViD dataset for experiments, which are not very popular in comparison with Kinetics and something-something. As a result, the proposed approach could not compared with recent transformer-based approaches. Comparing the proposed transformer-based method to only non-transformer-based methods is not convincing in an age when transformer-based methods are breaking new records every day. I strongly suggest the authors to include comparisons to recent transformer-based methods in more datasets. \n- Impact of the number of tokens (S). S seems an important hyper parameters in the proposed method that makes trade-offs between speed and accuracy. It would be great if we can see its effects and the robustness on the choice of S. \n\nOverall Evaluation:\n- The paper proposed a tokenization approach for video representation learning. And, this is beneficial for video networks in terms of both speed and accuracy. But, I have some concerns on the technical novelty and the experiments, as discussed in Weakness. My current recommendation is on borderline slightly leaning toward accept. I would like to finalize my rating based on how the authors address my concerns during rebuttal.  ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I suggest the authors to discuss the limitations in the paper. \n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "Cy9quUyR8DM", "reviewer": "Reviewer_Lkvt", "summary": "The paper introduces a dynamic tokenizer that constructs the tokens using a set of convolutional layers, each selecting regions of interest. This is then combine with a space-time like attention mechanism. The idea is intersecting but important experiments are missing.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "On novelty:\nThe idea of generating tokens dynamically makes sense. There is really no particular good reason why one should allocate one token per patch irrespective of its content. This is why I found it surprising that the paper doesn't actually test this in a transformer, as in replacing the tokenizer from ViViT (the 3D tokenizer) and TimeSformer/ViT (2d tokenizer) with this. As is, this is a variation of a spatio-temporal attention layer that is thrown in the network which makes the novelty more incremental, considering both views may solidify this work.\n\nResults and ablation:\n- To follow up on my previous point what is the performance of ViVit and Timesformer when their tokenizer is replaced with the proposed one? (Note, that the FLOPs should ideally be ketp alike to avoid bias)\n- What is the accuracy of a network that exclusively uses this blocks ones (i.e. as in replacing the \n- The authors mention multiple times ViViT and TimeSformer - yet there is no actual tete-a-tete comparison against them.\n- Experiments on Kinetics are really a must if one want's to abe able to easily compare against the entirety of the state-of-the-art spectrum. Currently it's hard to properly understand how this approach positions itself in terms of accuracy in respect to the broader literature.\n- What is the computational cost of the methods listed in Table 1&2? \n-  L191 claims that this is a full space-time attention, motivating as such the reason why it outperforms disjoint space and time attention. Yet, in the supplementary material the claims are slightly different as the later actually outperforms the former. This is claimed to be due adaptivness. Given that a full space-time attention looks at all patches across space-time shouldn't this adapt too? Could the reason be due to overfitting of the space-time attention? This won't be too surprising as transformers already need much heavier augmentations to train well on images. As such, to what extend this results could be affected due to a) suboptiomal hyper-parameters and b)sub-optimal augmentations that may put to advantage one case over the other?\n\n\nOthers:\n- The provided training details are insufficient to reproduce this work. What are the augmentations used for example?\n\n- It was a bit unclear at first that the token learner is not intended to actually replace the tokenizer from transformers as portrayed early on in Section 2 when a parallel with ViViT and TimeSformer is made.\n\n- The paper is only 8 pages long (out the  9 pages limit), as such I am not sure why there was a need to create an appendix instead of attaching it to the main paper as most ot the results will fit well in the main paper.\n \n- L358 section number is missing", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The paper does a reasonable job to address this, perhaps a small comment: The conclusion lists \"One of the remaining challenges [...]\", what will be the others?", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "7"}, "extra_text": {}}, {"review_id": "3YVa-W9MKW0", "reviewer": "Reviewer_EDW8", "summary": "This paper introduces a method of adaptive space-time tokenization (TokenLearner) for video representation learning. Instead of using fixed/pre-trained tokens for videos, TokenLearner aims to generate multiple spatial attention maps for the architecture to better focus on regions with rich information, thus obtaining better tokens. The overall architecture mimics a combination of X(2+1)D and Bottleneck Transformer while inserting the TokenLearner module. The authors conduct experiments on Charades and AViD datasets, and the proposed method outperforms previous state-of-the-art models. The authors have also carried out ablation study on their design choices.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, the work is interesting, inspiring and promising. The proposed method is well motivated and investigated. The experimental results are impressive.\n\n**Originality**: To the best of my knowledge, the method is novel.\n\n**Clarity**: The paper is mostly well-written and easy to follow. Clarification on the method details is welcomed.\n\n**Significance**: The experimental results are promising, as TokenLearner achieves state-of-the-art performance on both datasets of Charades and AViD. It will be more beneficial to the community if the codebase and models can be released.\n\n**Strengths**:\n\n1. The method is well motivated. Video data are of large size and an efficient and effective model should adaptively focus on regions with rich information. The authors propose to enable such adaptive attention in the tokenization of videos.\n2. The experimental results are impressive. On Charades, without pre-training, the proposed method outperforms previous state-of-the-art methods. On AViD, the proposed method outperforms prior work by a significant margin.\n3. Exhaustive ablation studies are conducted, including different space-time transformers, different model components, and alternative designs to TokenLearner.\n4. Visualization of the spatial attention maps looks convincing.\n\n**Weaknesses and Questions**:\n\n1. The function $\\alpha_i(\\cdot)$ is very light-weighted as it is only a single convolutional layer (or a MLP layer) followed by a sigmoid activation (L79-80). However, I wonder if a single learnable layer is sufficient for the job of generating various and diversed attention maps. Have you tried larger models for $\\alpha_i(\\cdot)$?\n\n2. My understanding is that the adaptive tokenization only happens on the spatial dimension, as there are only spatial attention maps learned in the tokenization. Have you tried similar approach on the time dimension?\n\n3. Computation in the section 2.2 is not very clear to me. What is the shape of $f_q(z_i)$, $f_k(z_j)$ and $\\gamma(f_q(z_i) \\odot f_k(z_j))$? More mathematical details are needed in L91-100.\n\n4. How many crops are used in Charades evaluation? Prior work shows that the number of crops affects the final performance greatly, and I wonder if consistent hyperparameters are used in prior work and this paper.\n\n5. On Charades, the performance gain due to TokenLearner seems marginal (shown by comparing the last two rows in Table 1, mAP 58.5 vs 59.6). However, TokenLearner seems more helpful on AViD (shown by comparing the last two rows in Table 2, accuracy 50.1 vs 53.6). Any thoughts on why and what TokenLearner is actually helping with?\n\n6. Typos. \n- L63: \"TokenLearner\"\n- L91: shouldn't it be $|Z| = ST$?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have addressed potential societal impact in the section \"Broader Impact\".", "review_text": "", "extra_scores": {"ethical_concerns": "I don't find significant ethical concerns.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}]

Author rebuttal:

Meta-review: After the rebuttal period all reviewers rate this paper as being past the threshold for acceptance.
The authors did a good job of addressing questions of novelty during the rebuttal and additional experiments were well received by reviewers. The authors have promised to clean and polish some parts of the manuscript flagged by reviewers - please do this.

The AC recommends acceptance
Decision: Accept (Poster)