Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper presents a video model for few-shot action recognition. It has two main contributions: (1) NAS to search for model components from a Transformer search space w/ divided space-time attention (space: selecting Spatial/Temporal attention blocks, selecting attention heads in {6,12,16}), and (2) a Prototype Alignment module to compute similarity scores between query and support sets (similar to TRX [25]). Transformer search space is reduced during training based on a shrinkage strategy. The corresponding shrinkage score has the following intuition to determine the strength of an operator (i.e., S/T attention or different number of heads): If the expected loss of a subnet with a certain operator deviates (beyond a threshold) from the expected loss of subnets with the same compute budget, then the corresponding operator is dropped from the search space. Simply put, an operator is useless if it performs worse at the same budget. The authors validate their method with few-shot settings in UCF101 and HMDB51, but mainly consider models initialized from scratch (w/o any ImageNet pretraining) in contrast to previous work. Ablation studies further provide insights into different components of the proposed design.
strengths_and_weaknesses: Strengths:

- This paper presents a well-motivated idea and sound arguments in general. The selection of spatial/temporal components makes the design simple, and the search more convenient. It makes sense to vary the number of temporal components in settings w/ different number of input frames.

- The claims of the paper are thoroughly validated in the experiments. Also, the provided ablations give insights into each component in the proposed design.


Weaknesses:

- The pretraining setup that the authors follow seems to be unfair. In fact, the authors initialize the models from scratch rather than ImageNet-pretrained weights (in contrast to previous work), citing that there is a possible information leak from the pretraining data to few-shot classes. I do agree with this comment. However, the baseline methods that the authors compare against have significantly-larger budget (params/FLOPs) and they may be at a disadvantage compared to the proposed method, if not pretrained on a large-enough dataset. Because, large models tend to overfit more on small training datasets. It would be interesting to see the performance w/o any pretraining at a similar compute budget. In Table 7, authors pretrain on UCF101 which is still a smaller dataset.

- The proposed Prototype Alignment module is well-motivated and a main contribution of the paper. However, I wonder if this is truely novel. It has similarities with the proposal in TRX which also presents a query-specific class prototype for computing distance. Also, the authors call this module 'spatio-temporal alignment' (Section 3.4), when in fact, there is no spatial information at the input to the module. This only performs temporal alignment.

- Some information missing regarding the shrinkage strategy (can be addressed with another equation). It says that an operator is discarded if its score 'deviates' from the expected score at a similar budget. The score could be significantly lower/higher in both directions beyond the threshold, and a lower score should be better as far as I can see. Is it not the case? This should be clarified.

- Writing in the paper is sometimes missing critical information and hard to follow. It can be improved. Make sure the paper is self-contained (eg: L113: 'as in [15]').

- Specific hyperparameter settings are missing, especially in the shrinkage algorithm, which is probably important to include/discuss.


[Post-rebuttal]
- Authors have responded to all my concerns and I am  satisfied with their answers. 
questions: - Why is this the best search space? This is not well-motivated. I believe the authors should discuss why these dimensions (spatial/temporal/heads) are the ones to search in and whether it is sufficient. There are many other aspects such as spatio-temporal resolution, channel-expansion, parallel/multi-path connections, components beyond self-attention. I agree that including all these axes will be practically impossible, but the choices of authors should be well-motivated compared to other options.

- Is the inequality in Eq. 5 correct? I think one should be reversed. Is it not true that when the budget is higher the loss is lower in general?

- I do not fully understand the definition of a subnet based on its operators (in L125). I see the motivation, but the notation seems off. Why use summation for each operator within a layer but union across layers? Also, what do you mean by saying the sum of all indicator functions in a layer equals to 1? Please clarify this and better represent it in the paper.

[Post-rebuttal]
- Authors have clarified my questions and I am  satisfied with their answers. 
limitations: No major negative societal impact.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes a spatio-temporal feature matching and alignment method for the task of few-shot action recognition with the end-to-end design of models. To achieve this aim, the authors construct a transformer space with spatial and temporal attention elements and search for the optimal Spatio-Temporal representations from few-shot videos from the architecture perspective. In detail, this paper introduces a transformer space shrinking strategy to evolve transformer space and speed up the architecture search adaptively. Besides, this paper proposes a more efficient and general spatio-temporal prototype alignment method, which can be conveniently adapted to arbitrary length video matching. This paper is novel, and the promising results compared to baseline methods prove the efficientness of the proposed method.
strengths_and_weaknesses: + The formulation of transformer space shrinking is good. It is interesting and reasonable that this paper leverages the expectation of subnets and budgets for evaluating different operations. However, this method is a little complicated. Do authors have any plan to open source the code for the contribution of the community?
 
+ The authors propose a spatio-temporal prototype alignment method. I am considering the efficiency of this new method. Does the proposed method have the same computation budget as the old way?
 
+ The visualization of searched architectures is very impressive, and the search architectures' analysis seems promising.
 
+ With much smaller FLOPs and parameters, this algorithm achieves relatively good performance compared to the baseline methods.
 
- This paper only provides the experiments without pre-trained weights. I am curious about the results of the pre-trained weights with the ImageNet dataset. With this setting, can this algorithm still achieve a promising result?
 
- Fig 4 and Fig 5 analyze the effect of search space shrinking from the aspect of supernet. I notice that the test loss in Fig 5 starts from 2.5 epochs. Why not show the results from the 0 epoch? Besides, I do not think there is a direct correspondence between the training, testing, and removing useless operations. The supernet may get better training because of the smaller search space rather than removing redundant ops.
questions: As illustrated in the main review, I curious more about the code, computation budget, and some ablation studies of this algorithm.

limitations: Yes, the authors adequately discussed the limitations of this paper.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper presents a neural architecture search method for few-shot action recognition. The main contribution is the proposal of transformer space shrinking strategy and spatio-temporal prototype alignment. Experiments are conducted on HMDB51 and UCF101.
strengths_and_weaknesses: Strengths:
1. Ablation studies are conducted to evaluate the effectiveness of each component.
2. The proposed transformer space shrinking strategy reduces the time and cost for architecture search.
Weaknesses:
1. I understand that there may be some contradictions between pre-training and few-shot learning. However, since almost all of the state-of-the-art methods are based on pre-trained weights instead of random initialization, the authors should also report performances with pre-trained weights to ensure comparability among them.
2. To demonstrate the superiority and generality of the proposed method, experimental evaluations should be performed on large-scale datasets of action recognition, such as Kinetics and Something-Something, just like the competitors did.
3. Some closely related works, such as [a, b], should also be introduced and compared in the paper.
[a] Xiatian Zhu, Antoine Toisoul, Juan-Manuel Perez-Rua, Li Zhang, Brais Martinez, and Tao Xiang. "Few-shot action recognition with prototype-centered attentive learning." In BMVC, 2021.
[b] Shuyuan Li, Huabin Liu, Rui Qian, Yuxi Li, John See, Mengjuan Fei, Xiaoyuan Yu, and Weiyao Lin. "TA2N: Two-Stage Action Alignment Network for Few-Shot Action Recognition." In AAAI, 2022.

questions: 1. Performance comparison with the state-of-the-art methods when using pre-trained weights
2. Experimental results on large-scale datasets
limitations: See above
review_title: null

Meta-review: metareview: All three reviewers lean towards the acceptance of the paper. The reviewers believe the rebuttal has addressed their concerns. The AC recommends acceptance of the paper, and suggest the authors to include the materials and the discussion they promised in the rebuttal in the final version of the paper.
confidence: Certain
Decision: Accept