Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The method proposes a module to dynamically scale the tensors within a temporal action recognition model that can act as a drop-in addition to existing models. To achieve this, the per-frame input tensors are pooled into vectors, and a LSTM is used to condition the linear scaling of the tensor depending on the (avgpooled) representation of the current frame, but also previous predictions. This is also extended to include the predictions from the previous layer. There are also a couple of tweaks to the LSTM to improve performance.
main_review: The paper becomes somewhat clear as you read through, but it is not easy to get into. The terminology and the way the explanations go don't let the reader get an early understanding of what it's been tackled. For example, reading through the abstract I honestly had no idea what the paper was actually proposing. Then there are a number of explanations and terminology that are not easy to get through their meaning (I still don't understand what's "meta" about the method). 

The paper is somewhat novel. The mechanism used is indeed novel, but similar self-referential scaling modules are common for images and there are definitely multiple variants for video (something for example noted in the literature review). While the authors propose an LSTM-based methodology, some other works have proposed convolution-based approaches, but the global aim looks similar to me. Speaking in an abstract (a bit hand-wavy), you have a function that takes a sequence of input tensors and predicts linear scaling. That function, in this case, is an LSTM variant. Despite this, the variant is a valid one methodologically speaking, and in my opinion the approach is correct.

From the experimental point of view, experiments need to be more conclusive (and I know there are a lot of tables!). The main problems are a) On Kinetics400, the performance improvement seems to go down as the method is better (TSN+Ours is 2.7% better than TSN, but TSM+Ours is +1.1% better than TSM, and TDN+Ours is 0.71% better than TDN). b) I miss a more direct comparison with other methods with similar aim - I know the module improves over base architectures, but there are other alternatives to combine the base architectures with (e.g. section "action recognition with attentive models" lists a few). In order to be a useful module, you not only need to improve the base architecture, you also need to present an advantage over similar modules. c) FLOPs are ok (and the numbers look good), but timings would make a lot of sense as these modules are oftentimes quite sequential in nature and FLOPs do reflect particularly poorly what the impact is in practice (admittedly, this is not something common in the literature so it is hard to hold it against the authors).

Small details (no need to address them, just suggestions):

The ssv1/v2 table on the supplementary shoudl be upgraded into the main body, and maybe Table 4 is spurious and can be replaced instead?

The authors might want to revise a bit Efficient Spatial-Temporal Modelling. Some assertions do not sound great to me, e.g. l77 (mixing 2D and 3D convs) and l78 (two-paths networks are not designed for efficiency - slowfast adds the second path on top of a standard, relatively heavy backbone, so it is "efficient" in the sense that it doesn't add "too much" for the accuracy gain). 

Why is the "meta" term used? wouldn't it make the paper easier to understand to drop it?

The paragraph starting on l209 is not particularly clear when explaining the positioning of the modules. Maybe the authors could revise this paragraph to make it easier to understand (Later the ablation study has a very good explanation, which is the one I used to understand it)


limitations_and_societal_impact: The authors do not discuss the limitations despite claiming to do so in the introduction and supplementary. In terms of ethical impact, they reply N/A and might be correct.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper presents a Dynamic Normalization and Relay (DNR) module to augment the spatial-temporal representation learning for action recognition. The proposed DNR module can ease the effect of small batch size for video action recognition therefore improve the performance.
main_review: Strengths:
- The authors proposed a DNR module to dynamic normalize each layer according to the input sample, which can ease the weakness of small batch size when training a video action recognition networks.
- Extensive  ablation studies are conducted to show the effectiveness of their method.

Weaknesses:
- How the standardization parameters \mu and \sigma  are calculated is not very clear in the paper, is that the same as BN?
- The runtime of their method compared with other methods should be given apart from FLOPs, because they use LSTM structures in their module, FLOPs cannot represent the true performance of the model
limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes a novel dynamic batch normalization and relay (DNR) approach regarding the sequential information in videos. The authors have demonstrated extensive experiments and shown the effectiveness of the proposed DNR for a number of advanced backbones for video action recognition.

Apart from some technical confusions and questions, it is an interesting paper and the idea of dynamic batch normalization may be useful to the community.
main_review: This paper proposes a novel dynamic batch normalization and relay (DNR) approach regarding the sequential information in videos. The authors argue that traditional batch normalization is not frame-adaptive and ignores dependencies of the estimated normalization parameters between neighboring frames. Then the proposed DNR contains cross temporal DNR (CT) and cross layer DNR (CL) to handle the above problems. 

The paper is well presented and the idea of DNR for video action recognition is novel. The authors have demonstrated extensive experiments and shown the effectiveness of the proposed DNR for a number of advanced backbones for video action recognition. Apart from good results in the paper, I have few technical questions on the paper below:

1. P2 L54 mentions that the normalization parameters are generated dynamically both on training and inference. The authors also use the word “meta learning” in the abstract. Traditional meta learning requires extra training or fine-tuning procedure during the inference. However, I did not see that specific description on extra training on inference for specifying normalization parameters. I suppose inference stage directly uses the trained parameters and does not contain extra training during inference. If this is exactly what you are doing, I think the word “meta learning” and meta LSTM is somewhat misleading.

2. P7 L256 “relay distance is the number of sampled frames”. What does this mean? Does this correspond to the num_goups in _TBatchHyperNorm class in the provided codes? 

3. P8 L305 “We compare DNR with dynamic normalization (without relay) in Table 1d.” Does no relay mean that (e.g., for CT) operate batch normalization simply on different frames independently? 

4. Why not also perform three crops evaluation on SthV1 & SthV2 ? I understand 1 centre-crop is more efficient but three crops evaluation is able to give more convincing results to the audience.

5. It is good to see DNR introduces very limited extra FLOPs to the model in Table 6. How about the inference speed compared to no DNR? In some real applications, inference speed is more important than the introduced FLOPs. 

6. In table 3, the authors experiment with different batch size performances. Is the [linear scaling rule](https://research.fb.com/wp-content/uploads/2017/06/imagenet1kin1h5.pdf) used in this experiment since the different batch size requires different learning rates? The authors should clarify the learning rate used for each batch size experiment. 
limitations_and_societal_impact: Have not addressed adequate limitations and potential negative societal impact.

It can be done such as by providing the inference speed of with and w/o DNR to demonstrate how much latency is introduced to the inference stage. Also, there are a number of hyperparameters in DNR which may cause differences on different backbones.
review_title: null

Meta-review: This paper presents work on action recognition.  The main contribution is a drop-in module that dynamically scales tensors.  The initial reviews pointed to concerns over clarity, novelty, and experimetns.  The reviewers engaged in discussion and were satisfied with the authors' responses to their initial concerns.  The reviewers unanimously recommended acceptance of the paper after the discussion period, based on the novel approach to dynamic scaling for videos.  The authors are encouraged to make changes to presentation to help in describing the contributions in a clearer fashion, especially earlier in the paper, to improve its readability and future impact.

Decision: Accept (Poster)