Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper proposed a transformer-based architecture for online action detection. The architecture contains a long-term memory that stores information from the past 8 minutes and a short-term memory that processes the recent 8 seconds. Some techniques are used to reduce the complexity of transformer operations. The proposed method gets good results on THUMOS’14, TVSeries and HACS datasets. 
main_review: Using transformers to model long-short term temporal information is a novel architecture design, and I think the attention mechanism is suitable for retrieving information from long-short term memories. The proposed memory compression method for complexity reduction is a useful tool for long-term video modelling.

Overall, the experiments are well-conducted to show the effect of design choices. The analysis of runtime in the supplementary material is valuable for future research to follow. 

The submission is clearly written. As a suggestion, Fig 1 can be made more concise by avoiding repeating the video stream. 

The proposed LSTR model achieves state-of-the-art results on online action detection tasks on three common datasets, showing the effectiveness of the transformers and the long-short term design. 

limitations_and_societal_impact: Generally, the paper is good. Two limitations (suggestions) below:

* In the online inference setting (section 3.6), the current visual feature predicts actions conditioning on both the short and long memory. But in section 3.5 (Training LSTR), it is said a directional attention mask is used such that each element in the short-term memory (current -- 8s ago) can have an action prediction. In the training case, the earlier visual features are having less short-term memory, e.g. the earliest feature at 8s ago is not utilizing any short-term memory at all - which is different with inference. I wonder if this directional attention-based training setting is the most efficient. 

* The purpose of the first transformer decoder unit is to apply a dimension reduction for the long-term memory. Have authors tried other simpler alternative designs, like a projection layer?
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This work proposes a two-stage processing of video frames by Transformer decoders for video classification.  In the first stage a set of learnt queries “decode” a sequence of fixed-number of earlier video frames. Then in the second stage, features from recent frames act as queries and further “decode” the vectors output in the first stage. At the end of the second stage, final classification is carried out. 

The main contribution is this two-stage design of the video classification model.
The fixed-length “long-term” context of video frames is shown to be useful for “online” frame classification (i.e., w/o access to the future frames).

This work presents a set of experiments for different design choices, which is also informative.

The key experiments are online video classification on — THUMOS’14, TVSeries, and HACS datasets, where the proposed model outperforms the baselines reported in this work. 
main_review: ### Originality
The use of slot-based video features for long-term memory and its combination with short-term features using attention-based non-local blocks was introduced in long-term feature banks [28]. However, this work proposes a specific architecture using Transformer Decoders which is novel.
In the online action “detection” space, this method achieves strong results.
&nbsp;

### Quality
The design choices are validated through a good set of experiments. However, key weaknesses include: (i) use of non-standard datasets (however, these datasets seem to be standard for “online action detection”), and (ii) unclear if baseline methods were matched in capacity/modality etc. (see detailed comments below). 
&nbsp;

### Clarity
The writing is concise and easy to understand. However, there are a few potential errors (e.g., (3) in “clarifications/typos” below).
Significance
Given that the using slot-based video features for long-term context, and absence of results on more popular/standard datasets might limit the impact of this work.
&nbsp;

### Strengths
The proposed method is simple/easy to understand and implement. 
A number of experiments validate the various design choices.
Strong results are achieved by the proposed model on a number of popular “online action detection” benchmarks.
&nbsp;

### Weakness
1. The proposed model has a fixed context length (although this can be large).  This means that information from frames older than a certain length ($m_L + m_s$) do not influence the current decision. In LSTMs, for example, this context-length is continuous and not fixed (at least in theory). Hence, the model design does not accurately represent the name of the proposed model “LSTR”. 

2. On L268, an experiment is performed with recurrent models to compensate for this limited context length. However, it is found that this has no benefit over long LSTR memory sizes. This could be a limitation of the tasks in the experiment / difficulty in training the RNNs. 


3. A second limitation is that experimental comparisons against baselines are provided under non-standard settings. For example, it is not clear whether the baselines (e.g., LFB [58]) have a similar number of model parameters, model depth/backbone, modality ( i.e., two-stream (RGB + flow) or not), and inference time.  If the authors can add this information to the tables 1(a,b) and 2, it would help put the proposed method in context.


4. Further to the point above, a main reference model is the LFB [58], as it similarly has slot-based long-term memory.  However, the proposed work does not compare to LFB in the settings explored in the original LFB paper (e.g., Charades / EPIC-Kitchens – i.e. action recognition without localisation).


5. The comparison against “standard Transformer Encoder units” is provided on L297-L303. However, no comparisons are provided against standard Transformer **decoder** units, which are what the proposed model is based on. Comparison against a stack of Transformer decoder units (with learned query vectors, very much like the model proposed in [28]), without the late-stage fusion with recent features will be informative. This is a key ablation.


6. While the use of Transformers with “two-stage” video memory is novel, it is not very different in principle from the LFB [58] model, which also uses attention-based non-local blocks for fusing short-term and long-term features.   This might limit the impact of the proposed work.


7. THUMOS’14, and TVSeries are relatively small datasets (although popularly used in “online action detection” works), as compared to more “standard” datasets, like EPIC-kitches, Charades, AVA etc., which might further limit the impact of the proposed work. Authors are encouraged to adapt their method to the above datasets for greater impact.
&nbsp;

### Clarifications / Typos:
1. L79:  “most of these work only performs …” → these works … perform.


2. L92: what is ‘t’ in (txC) dimensional temporal sequence? Did you mean \tau instead?


3. L159: Unclear how $O(n_0 * (m_L +C))$ is “sub-linear w.r.t $m_L$”; clearly it grows linearly with $m_L$. Please clarify. 


4. L166: Saying $m_S$ are “output tokens” can be improved, by using the term “queries” instead.


5. L173:  Not having to do temporal unrolling / BPTT has more to do with the fact that the proposed architecture drops features older than $m_L$ time-steps (due to the FIFO queue), than using Transformers. 

6. Sec. 3.5:  Training for all frames in the short-term memory implies that the context length is shorter for the earlier $m_S - 1$ frames – this may be beneficial due to regularization, however, this point should be discussed.

limitations_and_societal_impact: No.  

1. Potential applications in online video classification (e.g., surveillance).
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper addresses an interesting and practical problem in videos – online detection of actions by examining only frames available till the present time and without any access to future frames. The problem is not only challenging but has practical significance too as online action detection is important in autonomous driving or surveillance scenarios and handling this with fast and light memory-weight training and inference enables one to address the problem real-time. The solution approach is based on the intuition that recent frames are more instrumental about the ongoing action while frames from extended past offer potential context to the present action. While differential exploitation of recent and extended past can definitely help (compared to using frames only from the recent past), it comes with extra computation and memory overhead. Based on the recently popular transformer philosophy, the authors came up with an encoder-decoder architecture with short and long term memories for recent and extended memories respectively. While related works have explored long and short term memories, the integration of the two outputs from them is not explicit and rather ad-hoc (pooling, concatenation etc.). The proposed work, on the other hand, made intelligent use of transformers with a two-stage memory compression design based on transformer decoders. Such a design is shown to be not only effective but also both memory and compute efficient. The experiments are well-designed. Extensive comparison with SOTA approaches and well-thought ablations show its superiority and robustness of design choices. The writing is crisp and adequate with insightful experimental analysis. The figures are also self-explanatory with descriptive captions. The related works are also covered good.
main_review: [Strengths]:
1. The paper addresses a very practical problem and proposes an efficient transformer based architecture for the purpose.
2. The experiments are well-designed and executed showing the superiority of the proposed approach.
3. The paper is well-written, the figures are well-illustrated and related works are well documented.
4. The long and short term memory with two-stage memory compression design based on transformer decoders for online action detection is novel and effective.

[Weaknesses]:

Though I am writing these under the heading ‘weaknesses’, they are mostly my queries for a better understanding.
1. Line 69-71 mentions about existing non-explicit integration of long and short term features using simple pool or concat operations. An ablation experiment with such heuristics for the integration would have been good.
2. Though online action detection is a very good usecase, is it possible to show the applicability of LSTR's superior modeling of long videos in other video applications?
3. Figure 1 (since the figure is said as best viewed in color): Does the different arrow colors mean anything?

In short, the paper seems to have addressed a practical problem with well-supported experiments and analysis and is also well written. Thus, I am in favor of accepting it at the moment.
limitations_and_societal_impact: It is described adequately in section 4.6 and possible future exploration of the temporal modeling capability of LSTR to both the spatial and the temporal domains is planned.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper presents a novel long video model named long short term transformer, with online action detection as the study case. The proposed transformer uses the encoder to model long-term historical information and uses the decoder to model the short-term information i.e. a few frames right before the current time. Experiments have been done on three standard benchmarks (THUMOS, TV, HACS) and the performance gains are clear compared to SOTA. Detailed analysis has been conducted as well.
main_review: Strength:
1. The paper is well written and organized.
2. The problem of online action detection is important in real-world applications and the proposed idea is interesting in the sense of being capable of dealing with very long video by decoupling long and short memories.
3. This model outperforms all of the sota methods on three benchmark datasets with clear gains, confirming the effectiveness of the proposed approach. Notably, the paper adopted LFB and tried to reproduce it on the THUMOS dataset for comparison.
4. The ablation experiments are extensive and can demonstrate the advantages of each component of the LSTR architecture.

Weakness:
1. The time complexity analysis in Sec 3.3 and Sec 3.6 is concrete and direct, however, there are no related experiments to evaluate the reduced time complexity in the paper. Although the runtime is shown in the Appendix, it is highly recommended to include the results in the experiment section since one important novelty of this model is its efficiency (as claimed in line 9 and line 40). Also, it is better to compare the efficiency of the memory compression with the single-stage design.
2. The idea of compressing historical information and decoupling long and near term information is not new for temporal sequence modeling. “Compressive transformers for long-range sequence modelling. ICLR 2020.” has similar motivation. I do acknowledge there are significant efforts in adapting the similar idea to online action detection, such a specific topic. But it's worthwhile mentioning and discussing these NLP works about dealing with long sequences.
3. It is nice that the paper compares with LFB [58] on THUMOS’14 because the non-local block can also aggregate features based on the attention operator. But in addition to TH’14, it will better convince me if there are comparisons with LFB on TVseries and HACS datasets as well.
4. This work only focuses on online action detection - how about offline temporal localization? In other words, I wonder if this kind of long short term modeling might also be useful for offline scenarios such as modeling long-term dependencies of sub-events in a long event while at the same time capturing fine-grained, localized information from short yet key moments. If space allowed, it would be interesting to discuss for future work.

limitations_and_societal_impact: The authors have described the limitations of the model. As described in Sec 4.6, this model is not robust for tiny motion and a lower performance when the subject is very far away from the camera.
review_title: null

Meta-review: The LSTR model is original and well explained and explored in the paper. It was well received by all reviewers, and the rebuttal has addressed most of the concerns raised.

One outstanding issue is that the model is not applied to several of the current datasets and tasks of interest (e.g. AVA, EPIC-Kitchens) and this will somewhat limit interest in it. The authors are encouraged to address this in the final version of the paper.
Decision: Accept (Spotlight)