Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper introduces a method of adaptive space-time tokenization (TokenLearner) for video representation learning. Instead of using fixed/pre-trained tokens for videos, TokenLearner aims to generate multiple spatial attention maps for the architecture to better focus on regions with rich information, thus obtaining better tokens. The overall architecture mimics a combination of X(2+1)D and Bottleneck Transformer while inserting the TokenLearner module. The authors conduct experiments on Charades and AViD datasets, and the proposed method outperforms previous state-of-the-art models. The authors have also carried out ablation study on their design choices.
main_review: Overall, the work is interesting, inspiring and promising. The proposed method is well motivated and investigated. The experimental results are impressive.

**Originality**: To the best of my knowledge, the method is novel.

**Clarity**: The paper is mostly well-written and easy to follow. Clarification on the method details is welcomed.

**Significance**: The experimental results are promising, as TokenLearner achieves state-of-the-art performance on both datasets of Charades and AViD. It will be more beneficial to the community if the codebase and models can be released.

**Strengths**:

1. The method is well motivated. Video data are of large size and an efficient and effective model should adaptively focus on regions with rich information. The authors propose to enable such adaptive attention in the tokenization of videos.
2. The experimental results are impressive. On Charades, without pre-training, the proposed method outperforms previous state-of-the-art methods. On AViD, the proposed method outperforms prior work by a significant margin.
3. Exhaustive ablation studies are conducted, including different space-time transformers, different model components, and alternative designs to TokenLearner.
4. Visualization of the spatial attention maps looks convincing.

**Weaknesses and Questions**:

1. The function $\alpha_i(\cdot)$ is very light-weighted as it is only a single convolutional layer (or a MLP layer) followed by a sigmoid activation (L79-80). However, I wonder if a single learnable layer is sufficient for the job of generating various and diversed attention maps. Have you tried larger models for $\alpha_i(\cdot)$?

2. My understanding is that the adaptive tokenization only happens on the spatial dimension, as there are only spatial attention maps learned in the tokenization. Have you tried similar approach on the time dimension?

3. Computation in the section 2.2 is not very clear to me. What is the shape of $f_q(z_i)$, $f_k(z_j)$ and $\gamma(f_q(z_i) \odot f_k(z_j))$? More mathematical details are needed in L91-100.

4. How many crops are used in Charades evaluation? Prior work shows that the number of crops affects the final performance greatly, and I wonder if consistent hyperparameters are used in prior work and this paper.

5. On Charades, the performance gain due to TokenLearner seems marginal (shown by comparing the last two rows in Table 1, mAP 58.5 vs 59.6). However, TokenLearner seems more helpful on AViD (shown by comparing the last two rows in Table 2, accuracy 50.1 vs 53.6). Any thoughts on why and what TokenLearner is actually helping with?

6. Typos. 
- L63: "TokenLearner"
- L91: shouldn't it be $|Z| = ST$?
limitations_and_societal_impact: The authors have addressed potential societal impact in the section "Broader Impact".
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposes a transformer-based neural network architecture for extracting video representations. Past approaches discretize entire videos into 3D (RGB-T) chunks as input 'tokens' to transformers. Instead in this work, an additional operation is proposed to extract a smaller number of tokens per frame. This leads to a more computationally efficient architecture that outperforms existing methods on the task of video classification. 
main_review: # Strengths

- The proposed idea of extracting limited number of tokens per frame is intuitive and easy to follow. 

- The attention mechanism proposed to extract fixed number of tokens is novel. Due to its simplicity it could be easily adopted in different computer vision applications. Considering the fact that transformers bring heavy computational cost in numerous domains, this idea could potentially be impactful. 

- Experimental results demonstrate that the model outperforms existing architectures for video classification on a wide range of multiple datasets. The ablative studies also show that the individual components used, VecT, TokenLearner and the Fusion operator, all contribute towards improve performance of the model. 

# Concerns

## Writing
- The abstract and introduction are not easy to follow. For example, the abstract begins by talking about space-time "tokens". To the best of my knowledge, this is not a term used commonly in computer vision. 
"The proposed approach is designed with the intention to allow the tokenizer to adaptively react to input video frames containing diverse visual content, and then to have the vector transformer and subsequent modules learn the underlying spatio-temporal interactions and long-range dependencies in video inputs." 
This sentence is extremely difficult to follow without any context provided. 
- Line 37: "Further, with these adaptively learned tokens, we apply a vector attention between key-query pairs." So far keys and queries have not been introduced and it seems like the reader is expected to know what "vector attention" is.  
- Line 38: "This can be thought as a version of Transformer in which the number of heads is the same as channels, allowing us to learn a different attention matrix for each channel." This also doesn't make sense without any additional information.

## Approach Details
- Line 91, why is $|Z| = S$ and not $ST$?
- Line 99, "the attention tensor is multiplied with the value matrix". Here the attention tensor is $ST \times ST \times C$ and the value matrix is $ST \times C$. So what kind of multiplication are you performing here? 
- Line 122-130, since your model is built on the X3D architecture, it would be useful to explain it in more detail here. For readers who are not well-versed with this literature, the rest of the description doesn't make sense. 

## Efficiency and Performance
- Comparing Row 4 and Row 7 of Table 3, it looks like your model is significantly more inefficient while only providing a ~1% gain in performance. In this light, what is the significance of the proposed architecture design choices?
- Line 202-Line 206, it seems like removing the learned tokenizer and replacing it with a predefined discrete grid only causes a drop in performance of ~0.8%. Again, here it doesn't seem like the TokenLearner is contributing much compared to this naive method. 
limitations_and_societal_impact: Yes, the authors have addressed limitations and potential impact thoroughly.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper introduces a dynamic tokenizer that constructs the tokens using a set of convolutional layers, each selecting regions of interest. This is then combine with a space-time like attention mechanism. The idea is intersecting but important experiments are missing.
main_review: On novelty:
The idea of generating tokens dynamically makes sense. There is really no particular good reason why one should allocate one token per patch irrespective of its content. This is why I found it surprising that the paper doesn't actually test this in a transformer, as in replacing the tokenizer from ViViT (the 3D tokenizer) and TimeSformer/ViT (2d tokenizer) with this. As is, this is a variation of a spatio-temporal attention layer that is thrown in the network which makes the novelty more incremental, considering both views may solidify this work.

Results and ablation:
- To follow up on my previous point what is the performance of ViVit and Timesformer when their tokenizer is replaced with the proposed one? (Note, that the FLOPs should ideally be ketp alike to avoid bias)
- What is the accuracy of a network that exclusively uses this blocks ones (i.e. as in replacing the 
- The authors mention multiple times ViViT and TimeSformer - yet there is no actual tete-a-tete comparison against them.
- Experiments on Kinetics are really a must if one want's to abe able to easily compare against the entirety of the state-of-the-art spectrum. Currently it's hard to properly understand how this approach positions itself in terms of accuracy in respect to the broader literature.
- What is the computational cost of the methods listed in Table 1&2? 
-  L191 claims that this is a full space-time attention, motivating as such the reason why it outperforms disjoint space and time attention. Yet, in the supplementary material the claims are slightly different as the later actually outperforms the former. This is claimed to be due adaptivness. Given that a full space-time attention looks at all patches across space-time shouldn't this adapt too? Could the reason be due to overfitting of the space-time attention? This won't be too surprising as transformers already need much heavier augmentations to train well on images. As such, to what extend this results could be affected due to a) suboptiomal hyper-parameters and b)sub-optimal augmentations that may put to advantage one case over the other?


Others:
- The provided training details are insufficient to reproduce this work. What are the augmentations used for example?

- It was a bit unclear at first that the token learner is not intended to actually replace the tokenizer from transformers as portrayed early on in Section 2 when a parallel with ViViT and TimeSformer is made.

- The paper is only 8 pages long (out the  9 pages limit), as such I am not sure why there was a need to create an appendix instead of attaching it to the main paper as most ot the results will fit well in the main paper.
 
- L358 section number is missing
limitations_and_societal_impact: The paper does a reasonable job to address this, perhaps a small comment: The conclusion lists "One of the remaining challenges [...]", what will be the others?
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: In this paper, the authors present an adaptive space-time tokenization for video representation learning. In the proposed method, each frame in a video is tokenized into a set of visual tokens through weighted summation over spatial locations. Then, vector transformer operates on a collection of tokens from entire video frames to temporally aggregate the information. The updated tokens are added back to the input tensor after linear pooling and broadcasting. The authors validated the proposed network module (TokenLearner) on two video action recognition benchmarks.
main_review: Strength:
- The idea of tokenizing frames instead of splitting into regular grids looks interesting and working well. The motivation is clear and the proposed approach is potentially beneficial to efficiency and performance of video recognition. 
- The paper contains comprehensive ablation studies that compares the proposed approach with possible alternatives. Visualization of tokenization is also provided.
- In general, the paper is written clearly and easy to follow.

Weakness:
- I have a little concern about the technical novelty. Learning an adaptive tokenization of an image is not new. In [A], a very similar mechanism for tokenizing an image into a set of visual tokens are proposed. While this work is proposed for image representation learning, extending it to temporal domain is straightforward. The difference is, in the previous work, they project back the information in the tokens after applying transformer. Projecting the information into the right location looks better than the global update by broadcasting. The authors should discuss this paper.
- The experiments are weak. The authors use Charades and AViD dataset for experiments, which are not very popular in comparison with Kinetics and something-something. As a result, the proposed approach could not compared with recent transformer-based approaches. Comparing the proposed transformer-based method to only non-transformer-based methods is not convincing in an age when transformer-based methods are breaking new records every day. I strongly suggest the authors to include comparisons to recent transformer-based methods in more datasets. 
- Impact of the number of tokens (S). S seems an important hyper parameters in the proposed method that makes trade-offs between speed and accuracy. It would be great if we can see its effects and the robustness on the choice of S. 

Overall Evaluation:
- The paper proposed a tokenization approach for video representation learning. And, this is beneficial for video networks in terms of both speed and accuracy. But, I have some concerns on the technical novelty and the experiments, as discussed in Weakness. My current recommendation is on borderline slightly leaning toward accept. I would like to finalize my rating based on how the authors address my concerns during rebuttal.  
limitations_and_societal_impact: I suggest the authors to discuss the limitations in the paper.
review_title: null

Meta-review: After the rebuttal period all reviewers rate this paper as being past the threshold for acceptance.
The authors did a good job of addressing questions of novelty during the rebuttal and additional experiments were well received by reviewers. The authors have promised to clean and polish some parts of the manuscript flagged by reviewers - please do this.

The AC recommends acceptance
Decision: Accept (Poster)