Searching for Better Spatio-temporal Alignment in Few-Shot Action Recognition
Authors: Yichao Cao
Xiu Su
Qingfei Tang
Shan You
Xiaobo Lu
Chang Xu
Venue: NeurIPS 2022 Accept (2022)
Keywords: Few-Shot Action Recognition
Temporal Alignment
Neural Architecture Search

Abstract: Spatio-Temporal feature matching and alignment are essential for few-shot action recognition as they determine the coherence and effectiveness of the temporal patterns. Nevertheless, this process could be not reliable, especially when dealing with complex video scenarios. In this paper, we propose to improve the performance of matching and alignment from the end-to-end design of models. Our solution comes at two-folds. First, we encourage to enhance the extracted Spatio-Temporal representations from few-shot videos in the perspective of architectures. With this aim, we propose a specialized transformer search method for videos, thus the spatial and temporal attention can be well-organized and optimized for stronger feature representations. Second, we also design an efficient non-parametric spatio-temporal prototype alignment strategy to better handle the high variability of motion. In particular, a query-specific class prototype will be generated for each query sample and category, which can better match query sequences against all support sequences. By doing so, our method SST enjoys significant superiority over the benchmark UCF101 and HMDB51 datasets. For example, with no pretraining, our method achieves 17.1\% Top-1 accuracy improvement than the baseline TRX on UCF101 5-way 1-shot setting but with only 3x fewer FLOPs.

Searching for Better Spatio-temporal Alignment in Few-Shot Action Recognition
Yichao Cao Xiu Su²; Qingfei Tang', Shan You', Xiaobo Lu', Chang Xu²t
School of Automation, Southeast University, 2School of Computer Science, Faculty of Engineering, The University of Sydney, 3 Enbo Technology Co.,Itd., China, *SenseTime Research
caoyichao@seu. edu. en, xisu59920uni. syydey edu..u, gingfeitang@gaail. com
youshandaensetime. com, xbluβseu. enu c.xudsydney adu.au
Abstract
Spatio-Temporal feature matching and alignment essential for few-shot action recognition as they determine the coherence and effectiveness of the temporal patterns. Nevertheless, this process could be not reliable, especially when dealing with complex video scenarios. In this paper, we propose to improve the performance of matching and alignment from the end-to-end design of models. Our solution comes at two-folds. First, we encourage to enhance the extracted representations from few-shot videos the perspective of architectures. With this aim, we propose a specialized transformer search method for videos, thus the spatial and temporal attention can well-organized and optimized for stronger feature representations. Second, we also design an efficient non-parametric spatio-temporal prototype alignment strategy to better handle the high variability of motion. particular, query-specific class prototype will be generated for each query sample and category, which can better match query sequences against all support sequences. By doing so, our method SST enjoys significant superiority over the benchmark UCF101 and HMDB51 datasets. For example, with no pretraining, our method achieves 17.1% 1% Top-1 I accuracy improvement than the baseline TRX UCF101 5-way 1-shot setting but with only 3x fewer FLOPs.
Introduction
One of the most important tasks in video understanding is to understand human actions, which is one of the representative tasks for video understanding [54]. In recent years, video understanding and action recognition have been witnessing remarkable progress with the emergence of high-quality large-scale video datasets [5, 15, 10]. However, this success relies quite heavily of manually labeled samples. The dataset annotation process is laborious, cumbersome, and time- consuming and restricts practical algorithm applications. Therefore, how to classify unseen action classes with extremely few annotated samples gives rise to the investigation of few-shot (FS) action recognition [52, 4].
large amount
Compared with the image task [49, 46, 48, 8, 50, 45], introducing temporal dimension makes the video task more complicated. For example, actions in videos often performed various speeds and occur at different time points. Further, action recognition usually needs to integrate multiple distinctive sub-action information to generate corresponding temporal representations, which are
"qqual contribution. Corresponding author.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

used for subsequent spatio-temporal feature matching, To address the few-shot action classification problem, most existing studies [52, 37, 3, 32, 39] use metric-based meta-learning framework to implement similarity comparison between query (test) and support (reference) videos. It first maps the input video to an embedding space via representation learning and then achieves the distance metric for comparing video similarities in an episodic task. These approaches follow the episodic training to meta-learn a backbone network [52] or temporal relational module [4, 3, 39J. However, we argue that these methods underestimate the importance of spatio-temporal representation, which is crucial for the basic concept of sew-shot action classification. And, there are some concerns about existing temporal relational strategies [39] on spatial features: (a) insufficient sparse sampling frames may limit the capacity for long-term temporal models, and (b) increasing the number of sampling frames will lead to the complexity of combination and matching strategy.
Inspired by these works, we present novel approach to explore the prime architectures as to sequence order of temporal attention and spatial attention blocks for the few-shot action recognition. To boost the search, we leverage a pre-defined transformer space that considers both temporal attention and spatial attention, and propose space shrinking strategy with the analysis of collected training losses during the search to explore the emphasis of spatial and temporal based information in different stages for the superior architectures in terms of accuracy and budgets. Finally, we propose nonparametric spatio-temporal temporal prototype alignment method to encourage each query video to match all reference videos in the support set for long-term temporal models.
The main contribution of this paper is summarized follows:
To explore the possibility of the optimal video understanding architecture for few-shot action recognition, we incorporate spatial and temporal attention elements in our transformer space. It allows the model to spontaneously choose and focus on different types of information at various stages, which is crucial for obtaining better spatio-temporal representation.
We introduce transformer space shrinking strategy to adaptively evolve Transformer space to speed up the supernet training and structure search. Based on this strategy, the time and cost for video understanding architecture search are drastically decreased.
We propose a more efficient and general spatio-temporal prototype alignment method, which can be conveniently adapted to arbitrary length video matching. Our method breaks the limits frames of standard relational modules for few-shot action recognition and achieves better capacity for long-term temporal models.
2
Related Work
Few-shot Image Classification. Few-shot learning [13, 25, 26] aims learn novel classes with only a small number of labeled samples. One of the fundamental tasks in FSL is few-shot image classification [28, 18], which is preferable using a combination of representation learning and a metric-based based classifier. The terminology "n-way k-shott means that models are used to classify n distinct classes with only & reference samples per class. This process also called "episode" [42]. Recent works often average the representation of a support class to calculate a more robust class prototype [18, 12]. CrossTransformin Transformers [1 I] employs spatial alignment between query- and support- set images to improve generalization for unseen classes. Matching- based methods [14, 30, 38, 43] tend to learn a deep distance metric or similarity functions to compare examples [7, 9, 20, 40].
Few-shot Video Action Recognition. Recent attempts [19, 54, 23, 51, 41, 56, 5] focus more on temporal alignment [55], which aims to align two video sequences in temporal dimension. Notably, [4] explicitly learns distance measure and representation independent of non-linear temporal variations in video sequences. [53] proposes prototype-cene centered contrastive learning loss and a hybrid attentive learning mechanism to minimize the negative impacts of outliers and promote class separation in few-shot action recognition task. [22] designs temporal transform module to handle action misalignment, which consists of two parts: a localization network and a temporal affine transformation. [24] formulates the distance between video sequences as transportation cost. Temporal Relational
(TRX) [27] operates over a different number of frames for higher-ordered temporal model and query-specific class prototypes construction. However, this method combines a small number of frames that may have limited capacity for modeling long-term dependencies. addition, as the number of input frames grows, the number of combinations increases sharply, which makes the processing of temporal relation module more complicated.

---

Neural Architecture Search (NAS). NAS [36, 33, 44] aims to automatically learn/search for better neural network architectures. Earlier, MetaQNN [1] is proposed search performing CNN architectures via reinforcement learning. [29] proves that evolutionary algorithms are capable of constructing large, accurate networks in a huge search space. Single Path [17] proposes to train each architecture alternately with two-stage search schema. AutoFormer [6] first adopted the one-shot
NAS framework for the VIT based architecture search.
3 Method
3.1
Problem Formulation
The goal of few-shot video action classification is to classify unlabelled query video into sev- eral new classes with only a few labeled video samples, referred to as the 'support set'. The dataset is separated into a training set D.. = {(r,, y4) yi Ctrain validation set Dvel {( ((..84) | E Cvalidotion j. and a test set Drest = {(z,, 31) yı E Cresth. where the validation set split from the training set with 1/10 samples, 24 the i-th video sample with class label yi- Different from the general supervised learning, few-shot training and test categories are disjoint, Le. Cirain Ctest = To make training more faithful to the testing method, we follow episodic training [27][47] to train the model. In each episodic task, training samples are randomly sampled from the training set n-way k-shot setting. Concretely, a support S and query set Q are sampled from Dr as S, Q = {I vi) ) 3 E Cupport where Caupport S Ctrain is the n selected classes. Compared with few-shot image classification, the introduction of the time dimension in video classification demands the model to be able to learn temporal patterns with limited sample size. Next, we describe our proposed method for few-shot video action recognition.
3.2
NAS for Few-shot Video Action Recognition
For a supernet [34] A with weight W and Transformer space A, each subnetwork (subnet) a inherits its weight w, from W. The optimization and search problem in our supernet training stage can be formulated as:
arg max ACC (a, (a), Duat)
aeA
s.t. W = arg Lce (wa: WA. a, Drr)
(2)
WA
where W a set of trained weights for the whole Transformer space A, fee and ACC denote the cross-entropy loss and the accuracy for each subnet a in the episodic task, respectively.
We build our overall Transformer space A with 4.74 1 subnets in Table 1. Since the video understanding model may need to adjust the layout of temporal and spatial attention in different layers for better temporal model ability, we design independent Space "SAB" and Time Attention Blocks "TAB" for our Transformer space. Although the video features are transformed by TAB and SAB, the output feature dimension is unchanged. To avoid the model being overly complex, We incorporate the dimension downsample block, named "Spatial Downsampling Block (SDB)" in Table as in [16]. The core operation of this block is use the down sampling spatial self-attention, which maps an input tensor of size (C, W. H) to an output tensor of size (C", W/2, H/2) with C' > C. Due to the change in scale, we can easily control the complexity of the model in different stages. We present more details about our Transformer space in section 3.5.
3.3 Transformer Space Shrinking
In the previous subsection, we design large Transformer [35] space covering variety of operations to explore the optimal network structure, However, this also causes some problems: 1) too large Transformer space increases the difficulty of training and optimization of supernet, 2) redundant operations in Transformer space increase the difficulty of structure search. How to design a variety of Transformer spaces for different parts is another challenging task. To this end, propose Transformer space evolving strategy. Towards given Transformer space, the unreasonable space parts automatically shrunk, and the whole space gradually converges
more compact state.
To evaluate each operation in Transformer space, we define score S(i, as the evaluation metric for j-th operation in i-th layer. Formally, any subnet E A can be expressed the result of
3

---

Stage (A) Layens
Stage 2 (N; Layens)
Stage (1g layent
Query
Time H6
Spacia Attn, H6
Timm Asto,
Space
00000000
EXCM
Time Alrr.
Time
Time
Ex
Support Video
Space
10000000
Spaco Temporal Algnment
IT.C, W//6, 8/16)
G. 8/332
IT. G/G4W 81/44
C,
Figure 1: Overview of the neural module be searched. The architecture of the network is determined by different operations per layers. There are three main stages in which space and time attention operations can be searched and selected.
Algorithm 1: Training supernet with Transformer space shrinking
Input: Supernet V with weight W and Transformer space A. maximum training epochs T, warm up epochs Two shrink epochs P, score threshold Thr and shrink percentage K.
Init 0;
while T
randomly sample subnets from supernet N: train one-shot supemnet with Transformer space A;
if 2 Tan then
record the loss and FLOPs of the subnet af, for each operation Ozyi
-T% = 0 then
calculate the S(i,j) for each operation with Eq. (6) and Eq. (7);
A + A.Shrink(K,Thhr
end
Output: The optimized weight WA for supernet N, and the shrunk Transformer space A.
layer by layer operation stacking. The subnet a can be represented as a = U, ES 1505,, where 1 (0, 1} denotes indicator function, which indicates whether o,, is selected in In each layer of supernet, subnet selects only one operation. Thus, for the indicator function 13 E {0, 1} in i th layer, the sum of the indicator functions 1: Σ, 1j = 1. The training loss for subnet a can also be expressed as
UI 1j00s/
Coss(a) Loss
Loe (Wa:NN. a, Der)
(3)
To represent the bi-directional relationship between operation o,, and subnet a, we denote the subnet which includes o.,j by a. For subnet with Otj it is with huge variance to directly measure the contribution of O,, the performance of a. However, the same operation will appear in many subnets, and we can transverse the search space and leverage the performance expectation of all these subnets a to evaluate the score of Oz Thus, we can obtain the relationship between score and training loss:
S(i.j) S (ois) & (aEU(A)),, foo (way;:Na J. D..)]
(4)
Considering that different operations have distinct effects the computing budgets of subnets, is often desirable to limit the computing budgets of subnets in practical scenarios. However, with the sufficient and same amount of data, those models with big computing budgets may have more advantages in persuing good results. If left unconstrained, the operation scores can be as
S ((oi,)) > S (o...): if EaEU(A), [B (a).m) < Ea [B (n)
(5)
where B(a?,) ) denotes the computing budget of subnet a To address this problem, propose modified approach considering the impact on the computing budget of subnets. We update Eq. (4) as
[
[Cos (wae, NA o. Der ] + (B
S(i.j) S (or,)) = EEE
(6)
where 5(.) is correction term related to the statistical expectation for subnet a By, denotes the statistical expectation for subnet a. We hope to correct the impact of different operations the

---

Query video
Twhd
Unner
Lineat
Lineaf
Linear
Linwar
Linear
((tg) 1000
MatMuu
MatMul
Scale Softmaa
Softmaas
f(Kg)
[oA,F
M (Fgi
Time Attention MatMuu
Space Atsention MatMuu
f(g)
Align
0000-
f (2))
[othT.cl [Twiu]]
Support video x,
time
Time Attention Block (TAB)
Space Attention Block (SAB)
Figure 3: Spatio Spatio-temporal Prototype Align- ment.
Figure 2: The essential operation of stack layer.
subnet FLOPs by 8(-), ((,,j). Then, the correction term can be defined as
accurately evaluate the relative performance of different opertions by
(B (E) [Cee (wa: a, Por)
Those high-scoring (low performance) operations that deviate from the average score by more than certain threshold will be dropped.
Specifically, we make statistics for all the operations after each P epochs during supernet training. We discard low-performing operations that deviate from the average score by more than certain threshold but not more than K. percent of the overall operation. In this way, Transformer space shrinking can be achieved in supernet training. As a result, the following search process can be carried out more efficiently on this shrunk space.
3.4 Spatio-temporal Prototype Alignment
Given query video F.g and support video v, and their embedded features fxxg), f(x)) RTx... a commonly used similarity measurement method
d (x.) TC. El1((e) (x
(8)
where ( ..)) is the similarity distance between video F. and v.. T and C, are the temporal length and feature dimension, respectively. The feature spacing between two videos is the mean value of the difference in temporal and channel dimensions. However, this direct measurement has disadvantages in few-shot video action recognition: it is complicated to match changeable actions performed at various speeds directly and occurred different time points in extremely few videos. To this end, we propose nonparametric apatio-temport feature prototype alignment method to better deal with feature matching in few-shot video action recognition. Specifically,
M (Ig, softmax [f (xig) f (a.))
where M(Ig, ..) denotes a frame-level attention map with dimension of T x T. This provides the possibility for us to calculate the query-specific prototype w.r.t the query Zyg such as
f (x.)= = M (Iig. I.) (x.)
(10)
where f(x.) is a temporal aligned support sample representation, as shown in Figure 3. The updated feature distance between the aligned support feature and query feature can be calculated as
d (Z.g = TC, Σ f (Ig)
(xx))
(11)
Then, this measurement method can be casily applied to the support set St of class i.c.
d Sx
Σ d (x),
(12)
PEII

---

Table I: Macro Transformer space of our SST model. Given an input video with dimension T x x W H, we search over operations and the number of heads at 3 stage. Within each stage, "Choice Block" indicates the search from blocks of "TAB" and "SAB". "SAB"."Spatia" "Spatial Downsample" represents the spatial downsampling block (SDB).
Stage
Layers
Operations
# Heads
Output Size
Patch Embedding
Convolution
T, C1, W/16, H/16]
Choice Block
(TAB, SAB)
6,12,
C1, W/16, H/
Spatial Downsample
SDB (TAB, SAB)
12
(T. T2C2, W/32, H [32
Choice Block
6, 1212, 16
[T, C2, W/22, H/32
Spatial Downsample
SDB
12
[T, Cs. W//44,H666
Choice Block
(TAB, SAB)
6,12,
[T. C3, W/64, H/64
Output
Norm
(T,Co)
where Tp denotes the support example in support set S During training inference, the query video Ty is assigned the class of the closest query-specifer prototype. This is an efficient nonparametric spatio-temporal prototype alignment method that can easily be applied two apatio-temporee feature similarity measurements. Generating an aligned query-specific prototype for every support video can obtain more reliable prediction results in the few-shot video action classification.
3.5 Transformer Space Design
To find the optimal combination layout for SAB and TAB, we construct transformer space for the FS video action classification. Let T x C' x W x H be the dimension of the input frames of our supernet. We extract multi-scale features hierarchy in every stage with a scaling step of from the input frames. These intermediate embeddings have spatial stride of 16, 32, 64 w.r.t the input. We do not reduce the time dimension to generate a feature representation with the same length as the input sequence. For each layer in our SST model, we search over the operation type and the number of heads in SAB and TAB. The model can not only choose what kind of attention to do in different layers, but also obtain the most appropriate number of self-attention heads.
Diagrams of our TAB and SAB are illustrated in Figure 2. where the w.h.c denote the width, height, and channel of intermediate embeddings. In the time attention block, a temporal self-attention mask with dimension wh x T x T is computed to capture temporal dependencies across frames. Similarly, spatial attention with dimension T x wh x wh can be obtained in the space attention block. Thus, these two modules have different functions in the process of video understanding. To facilitate the architecture search, these two modules share the same input and output dimensions. Table provides a basic overview of the Transformer space.
The input dimensions are both Twh x c,
4 Experiments
4.1 Datasets and experimental setups
Datasets. Our proposed method is evaluated on two popular benchmarks: UCF101 [31] and HMDB51 [21], using split from [27]. UCF101 [31] is human action videos dataset from Youtube, which contains 13320 video clips and 101 action categories. We used the split with 70/10/21 action categories in reiniing/alichaiionntttnnng HMDB51 [21] contains 6849 video clips divided into 51 classes. A split with 31/10/10 classes is used for HMDB51 [21]. The standard 5-way I-shot and 5-way 5-shot is applied to evaluate the performance of our method.
Implementation details. For each n-way k-shot episodic task, we randomly sample class with each class containing & examples as the support set. And we randomly sample one sample for each class in n classes the query set. Thus each episodic task contains x (k+1) samples. For video preprocessing, the sparse sampling strategy is used to fetch T frames for each video. During training, resize sampled frames to 256 x 256 and then randomly crop a 224 x 224 region as input. During testing, random crop is replaced by center crop.
For supernet, we construct the base Transformer space according to Table 1. The stage and block setting (the numbers of MHSA and FC layers) inspired by [2, 16]. Follow LeViT [16], the depth
of each stage is set to the same. And the head number h in the SAB and TAB is chosen within the

---

Table 2: Few-shot action classification results on HMDB51 [21].
5-way I-shot
5-way 5-shot
Method
Frames
Acc
Params
FLOPs
Acc
Params
FLOPs
TimeStormer [2]
8
33.2
40.7M
73.35G
41.7
40.7M
73.35G
TRX [27]
8
29.1
25.6M
41.43G
46.4
25.6M
41.43G
ARN [47]
20
45.2
60.6
4
39.2
8,54M
6.83G
57.1
8.53M
6.81G
Ours
8
51.1
8.89M
13.64G
60.4
8.91M
13.65G
12
52.4
8.87M
20.49G
62.2
8.86M
20.48G
Table 3: Few-shot action classification results on UCFI01 [31].
5-way I-shot
5-way 5-shot
Method
Frames
Acc
Params
FLOPs
Acc
Params
FLOPs
TimeSformer [2]
8
42.0
40.7M
73.35G
63.0
40.7M
73.35G
TRX [27]
8
46.7
25.6M
41.43G
67.0
25.6M
41.43G
4
60.1
8.61M
6.79G
68.2
8.63M
6.83G
Ours
8
63.8
8.87M
13.72G
69.7
8.84M
13.67G
12
65.4
8.76M
20.34G
70.4
8.87M
20.45G
set (6, 12, 16) to encourage each block to select its optimal head numbers freely. During training, the AdamW optimization method is used to train the supernet from scratch. All experiments are implemented with 8 Nvidia 1080Ti GPUs.
Furthermore, it should be mentioned that existing FS action classification all tend to rely on the pre-trained weights (e.g., pre-trained on ImageNet) to initialize the model. However, this pre-training may violate the basic assumption of few-shot learning that the query classes cannot be seen during meta-training [55]. We find that ImageNet contains very high-related classes those query classes during meta-testing, such as the "Guitar" class in ImageNet vs. "PlayingGuitar" class in UCF101. This makes ImageNet pre- training unreasonable and problematic. Thus, all model optimization in this paper is based on randomly initialized weights to ensure the fairness of the comparison results.
4.2 Comparison with state-of-the-arts
We compare our method with existing few-shot action classification methods such as ARN [47],
TRX [27] and existing video understanding methods such TimeSformer [2]. Table 2 shows the comparison of our method state-of-ther in 5-way -shhot and 5-way 5-shot problems on HMDB51.
As in Table 2, our searched models outperforms ARN [47], TRX [27] and TimeSformer [2] by large margin on the two protocols. Meanwhile, SST models have fewer parameters and FLOPs compared with previous methods. It is well known that processing more dense frames often means greater computing budgets. However, even if our SST processes dense 12 frames, it also has a good advantage FLOPs. From these comparison results, setting are generally higher than that of 1-shot. This is because more available video examples can help the model match the action pattern
also that the results of 5-shot
accurately in the inference process.
The results on UCF101 dataset are listed in Table 3. On UCF101 dataset, our method also outperforms the previous methods by a large margin. In particular, when the number of input frames is 8, SST improves TRX by 2.7% in 5-shot case and by surprisingly 17.1% % in 1-shot case. Note that the performance of SST with 4 input frames can exceed that of TRX with 8 frames, which demonstrates the superiority of our method in dealing with extremely few videos.
4.3 Ablation Study
Here, we systematically analyze the impact of different modules in our method. Specifically, we first show the effecctivnness of each component under the 8 input frames and 5-way 5-shot setting in Table 6. We follow the TimeSformer[2] and LeVit [16] to construct a baseline model, in which the intermediate modules are set manually in way similar to TimeSformer(2] These results show that the NAS can improve the baseline method with an improvement of 2.07%, which indicates the searched model has a stronger learning ability of spatio-temporal features. The proposed Transformer Space Shrinking and Temporal Alignment contribute to an improvement of 1.37% and 2.56%, respectively. All components collaborate and complement each other, reaching 60.42% finally. In what follows, we conduct several ablation experiments to evaluate the effectiveness of each module on HMDB51 carefully.
7

---

Table 4: Compare SST with non-searched models.
Table 5: Performance evaluation of TA.
5-way 5-shot
HMDB51
Method
Frames
Method
Frames
Acc
Params
FLOPs
5-way I-shot
5-way 5-shot
48.5
8.96M
6,94G
4
36.7
52.5
Plain model
54.6
8.96M
13,89G
w/o TA
8
40.2
58.0
12
56.3
8.96M
20.82G
12
46.6
58.9
4
57.1
8.53M
6.8TG
4
39.2
57.1
Ours
60.4
8.83M
13.65G
Ours
8
51.1
60.4
62.2
8.86M
20.48G
12
52.4
62.2
Table 6: Ablations of different modules of our method HMDB51 [21]. And we report the mean and std over 5 runs for SST.
NAS
Transformer Space Shrinking
Temporal Alignment
5-way 5-shot Acc
54.58 + 0.07
56.65 + 0.08
58.02 + 0.11
59.21 + 0.06
60.42 + 0.09
0.36
Method
Methed
0.35
Shrick
1.65
Loss Shrink
noShrink
potamak
1.80
Top-1 aceuracy
Test Ines
0.31
1.45
0.30
1.40
0.29
0.28
2.5
5.0
7.5
10.0
12.5
15.0
17.5 20.0
2.5
1.0
7.5
10.0
12.5
15.0
17.5
20.0
Apochs
Epocaa
Figure 4: The test accuracy of supernet.
Figure 5: The test loss of supernet.
4.3.1
The impact of NAS
To study the impact NAS, we conduct several experiments on hand- hand-engineered and searched architectures. In "Plain model", we follow the TimeSformer(2] and Le [16] to construct plain model, which has structure similar to our Transformer space, but the intermediate modules are set manually. According to Figure I, we manually set the alternating TAB and SAB modules in each stage build the final model. Then, the same hyperparameters are used to retrain these hand-engineered and searched models. As shown in Table 4, the searched architectures surpass the non-searched version by considerable gap while using fewer parameters and FLOPs. These results demonstrate the effectiveness of exploiting Transformer architecture in our method.
4.3.2 The impact of Transformer Space Shrinking
To better study the impact of Transformer space shrinking, we conduct two comparative experiments on HMDB51. We train a supemett on 1-shot task with 4 input frames. The average metrics of randomly sampled subnetworks are regarded as the metries of supernet. The top-1 accuracy and loss on test sets are reported in Figure and Figure 5, respectively. From figures, we can observe that the introduction the Transformer space shrinking strategy not only speeds up the convergence of the supernet, but also greatly improves the performance. Theoretically, the search space is wider, search time is longer, and training speed is slower, the stronger the need for space shrinking.
4.3.3
The impact of Temporal Alignment (TA)
Here we show the effectiveness of the proposed Spatio-tempon Temporal Alignment strategy. For "w/o Temporal Alignment", we directly measure the distance between temporal features with the dimension of T' x C. in the method of Eq. (8). For our complete method, support video features are first aligned to generated query-specific prototype and then compared with query features. For our complete method, support video features are first aligned to generate query-specific prototype and then compared with query features. From Table 5, we find the spatio-temporal prototype alignment improves the baseline various settings, which indicates the alignment strategy helps the FS video

---

#SAR
model setting
Se-4f
Fe-ar
1s-Br
1s-4f
1000
221 001
WOS
SH
STM RIS
CIM 0U
9TH OH
Exmple model under the "36-4f" setting
0Y5
SEL UP
WOS
SET ONS
DET 91
FEL SE
BOT
SH OVL
PV
GAL
6 N7AD
Stage
Stage
Stage
Exaple modd undar the "fe-
setting
Figure 6: Visualization of searched architectures.
Figure 7: Statistics under four settings.
Table 7: Comparison results using pretrained weights on HMDB51 [21].
HMDB51
Method
ImageNet-pretran
UCFI0I-pretrait Non-pretrain
Timeerormer [2]
48.3
41.7
TRX [27]
75.6
54.7
46.4
Ours
67.2
60.4
action similarity measurement. For example, when using 12 frames input, our method outperforms the non-aligned version by 5.8% on 5-way 1-shot setting. Furthermore, the proposed alignment operation is very efficient and does not introduce too much computational cost.
4.3.4 The impact of Pre-training
To prove the effectiveness our method using pretrained weight, we also pretrain model on UCF101 dataset and then finetune in few-shot learning way HMDB51 dataset. The results are listed in Table 7. All the methods are finetuned on 5-way 5-shot tasks with 8 input frames. As shown in UCF101 dataset.
table, the performance of three methods can be greatly improved after pretraining Among them, our method still performs better than the previous two methods. In addition, we also provide the results of TRX using Imagenet pretrained weight. It can be seen from the results that the performance TRX can also be greatly improved after pretraining with large-scale ImageNet dataset. However, stated in section 4.1, there is possibility that relevant information may be leaked to the unseen query class when sew-shot learning methods use pretrained weights. By pretrained with larger dataset, there may be UCF pretrained or randomly initialized weights, our method significantly outperforms the previous methods. The experiment in this subsection demonstrates that our proposed method is still better than the previous methods after pretraining to ensure that the model has better feature extraction ability.
severe class information leakage. In this work, whether using
4.3.5 Visualization
In addition to ablation experiments, visualize some searched architecture and statistics under different settings. In Figure 6, two searched structures are shown in detail, in which "5s-4f" denotes the searched model under 4 input frames and 5-shot setting, "TAB-H12" denotes TAB block with 12 self-attention heads, and so on for the other abbreviations. And Figure 7 shows some statistical results, in which the horizontal axis represents the three stages of four models, and the vertical axis represents the number of spatial and temporal blocks (positive to the SAB, negative to the TAB). can be seen from the results that although there are some differences the layout of TAB and SAB modules in the first two stages, the proportion of these two modules is relatively balanced. Surprisingly, we notice that the layout the third stage is relatively large, which is quite different from the design in current hand-crafted models. Obviously, our method can search for the appropriate and specific architectures for different input dimensions and different tasks. For example, for the same Shot task, the models select 7 TAB modules in the third stage when 4 frames are fed in. But when 8 frames are used as input, the models only select 3 TAB modules the third stage. We argue that the demands for the temporal modeling ability of the model are related to the time dimension sampling rate of the input data. When the model can obtain only fewer input frames, more temporal attention blocks will be placed in the third stage to improve the action modeling capability.

---

5 Conclusion
This paper introduced FS action recognition method for neural architecture search method with a Transformer space shrinking strategy and spatio-temporal prototype alignment. The discovered architectures can learn steadier frame-level representations from FS video samples. The spatio- temporal features are then utilized in prototype alignment operation for learning similarity between query and support actions. The query-specific class prototypes for each query sequence are generated to deal with various actions at different speeds and temporal offsets. An extensive set of ablations shows how optimal video Transformer architectures are searched, the benefits of Transformer space shrinking, and the importance of the spatio- temtooa prototype alignment. In future studies, will delve into the relationship between pretraining and few-shot learning. Avoiding the possible leakage of category information while using pretrained weights is still matter of research.
Acknowledgement
This work was supported in part by the Australian Research Council under Project DP210101859 and the University of Sydney Research Accelerator (SOAR) Prize.
References
[1] Baker, B., Gupta, O., Naik. N., Raskar, R.: Designing neural network architectures using reinforcement learning. arXiv preprint 1Xiv:1611.02166 (2016)3 3
[2] Bertasius, G., Wang, H.. Torresani, L.: Is space-time attention all you need for video understanding. arXiv preprint arXiv:2102.0509 2(3), (2021) 6, 8,
[3] Bishay, M., Zoumpourlis, G., Patras, L: Tarn: Temporal attentive relation network for few-shot and zero-shot action recognition. arXiv preprint arXiv:1907.0900 (2019) 2
[4] Cao, K., J., J., Cao, Z... Chang, C.Y., Niebles, J.C.: Few-shot video classification via temporal alignment. In: Proceedings of the IEFE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10618-10627 (2020) 1.2
[5] Carreira, J., Zisserman, A.: Quo vadis, action recognition? a new model and the kinetics datasct. In: proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 6299-6308 (2017)
1.2
[6] Chen, M., Peng. H., Fu, J., Ling, H.: Autoformer: Scarching transformers for visual recognition. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 12270-12280 (2021)* 3
[7] Chen, Y., Wang, X., Liu, Z., Xu, H., Darrell, T.: A new meta- basella for few-shot learning (2020) 2
[8] Cheng. Z., Su, X.. Wang. X., You, S., Xu, C.: Sufficient vision transformer. In: Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. pp. 190-200 (2022)
[9] Chopra, S., Hadsell, R., LeCun, Y.: Learning similarity metric discriminatively, with application to face
verification. In: 2005 IEEE Computer Saciety Conference on Computer Vision and Pattern Recognition (CVPR'05), vol. 1. pp. 539 546. IEEE (2005) 2
[10] Damen, D., Doughty, H., Farinella, G.M., Fidler, S., Furnari, A., Kazakos, E., Moltisanti, D., Munro,
J., Perrett, T.. Price. W., et al.: The epie-kitchens dataset: Collection, challenges and baselines. IEEE Transactions on Pattern Analysis and Machine Intelligence 43(11), 4125-4141 (2020)
[11] Doersch, C., Gupta, A., Zisserman, A.: Crosstransfommers: spatially-aware few-shot transfer. Advances in Neural Information Processing Systems 33, 21981-21993 (2020) 2
[12] Douze, M., Szlam, A., Hariharan, B., Jégou, H.: Low-shot learning with large-scale diffusion. In: Pro- ceedings of the IEBE Conference Computer Vision and Pattern Recognition. pp. 3349-3358 (2018)
[13] Fei-Fei, I.,, Fergus, R., Perona, P.: One-shot learning of object categories. IEFE transactions on pattern analysis and machine intelligence 28(4), 594-611 (2006) 2
[14] Garcia, V., Bruna, J.: Few-shot learning with graph neural networks, arXiv preprint arXiv:1711..00 (2017) 2
[15] Goyal, R., Ebrahimi Kahou, S., Michalski, V., Materzyyska, J., Westphal, S., Kim, H., Haenel, V., Fruend, I. Yianilos, P., Mueller-Freitee M., et al.: The" something something" video database for learning and evaluating visual common sense. In: Proceedings of the IEEE international conference on computer vision. pp. 1882258500 (2017)
10

---

[16] Graham, B., El-Nouby. A., Touvron. H., Stock. P., Joulin, A., Jégou, H., Douze, M.: Levit: vision transformer in convnet's clothing for faster inference. In: Proceedings of the IEEE/CVE International Conference on Computer Vision. pp. 12259-12269 (2021) 3, 7.8
[17] Guo, Z., Zhang, X., Mu, H., Heng, W., Liu. Z., Wei, Y., Sun, J.: Single path one-shot neural architecture search with uniform sampling, In: European Conference on Computer Vision. pp. 544 560. Springer (2020) 3
[18] Hariharan, B., Girshick, R.: Low-shot visual recognition by shrinking and hallucinating features. In:
Proceedings of the IEEE International Conference on Computer Vision. pp. 9018-3027 (2017)* 2
[19]
Hutchinson, M.S., Gadepally, V.N.: Video action understanding; tutorial. IEEE Access (2021)
[20] Koch, G., 7emel, R., Salakhutdinov, R., et al.: Siamese neural networks for one-shot image recognition. In: ICML deep learning workshop. vol. 2. p. O. Lille (2015)
[21] Kuehne, H. Jhuang. H., Garrote, E.,. Poggio, T., Serre, T.: Hmddb: a large video database for human motion recognition. In: 2011 International conference on computer vision. pp. 2556-245 2563. IEEE (2011) 6, 7, 8,
[22] Li, S., Liu, H., Qian, R., Li, Y., See, J., Fei, M., Yu, X., Lin, W.: Ta2n: Two-stad stage action alignment network for few-shot action recognition. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 36,
pp. 1404-1411 (2022) 2
[23] Lin, J., Gan, C., Han, S.: Tsm: Temporal shift module for efficient video understanding. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 7083-7093 (2019) 2
[24] Lu, S., Ye, H.J., Zhan, D.C.: Few-shot action recognition with compromised metric via optimal transport.
arXiv preprint wtXiv:2104.03720 (2021)3 2
[25] Miller, F.G., Matsakis, N.F.,, Viola, P.A.: Learning from one example through shared densities on
transforms. In: Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000
(Cat. No. PR00662). vol. pp. 464-471. IEEE (2000) 2
[26] Naik, D.K., Mammone, R.J.: Meta-neural networks that learn by learning. In: [Proceedings 1992] JJNN International Joint Conference on Neural Networks. vol. 1, pp. 437-442. IEEE (1992) 2
[27] Perrett, T., Masullo, A., Burghardt, T., Mirmehdi, M., Damen, D.: Temporal-relational crosstransformers for few-shot action recegnition. In: Proceedings the TEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 475-484 (2021) 2, 3, 6, 7,9
[28] Ravi, S., Larochelle, H.: Optimization as model for few-shot learning (2016)
[29] Real, E., Moore, S., Selle, A., Saxena, S., Suematsu, Y.L., Tan, J., Le, Q.V., Kurakin, A.: Large-scale evolution of image classifiers. In: International Conference on Machine Learning, pp. 2902-2911. PMLR (2017) 3
[30] Snell, J., Swersky, K., Zemel, R.: Prototypical networks for few-shot learning. Advances in neural information processing systems 30 (2017) 2
[31] Soomro, K., Zamir, A.R., Shah, M.: Ucf101: dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv: 1212.040 (2012) 6,
[32] Su, X., Huang, T., Li, Y. You, S., Wang. F., Qian, C., Zhang, C., Xu, C.: Prioritized architecture sampling with monto-carlo
search. In: Proceedings of the IEEE/CVF Conference Computer Vision and
Pattern Recognition, pp. 10968-10977 (2021)2 2
[33] Su, X., You, S., Huang, T., Wang, F., Qian, C., Zhang, C., Xu, C.: Locally free weight sharing for network width search. arXiv preprint arXiv:2102.0522 (2021) 3
[34] Su, X., You, S., Wang, F,, Qian, C., Zhang, C., Xu, C.: Benet: Searching for network width with bilaterally coupled network. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 2175 2184 (2021)3 3
[35] Su, X., You, S.. Xie, J.. Zheng, M., Wang, F., Qian, C., Zhang, C., Wang, X., Xu, C.: Vision transformer architecture search. Xiv e-prints pp. arXiv-2106 (2021) 3
[36] Su, X., You, S., Xie, J., Zheng, M., Wang, F. Qian, C., Zhang, C., Wang, X., Xu, C.: Vitas: Vision transformer architecture search. arxiv preprint arXiv:2106.1 13700 (2021) 3
[37] Su, X., You, S., Zheng, M., Wang, F., Qian, C.,, Zhang, C., Xu, C.: K-shot nas: Learnable weight-sharing for nas with k-shot supernets. In: International Conference on Machine Learning, pp. 9880-9890. PMLR (2021)2
[38] Sung, F,, Yang, Y., Zhang, L., Xiang, T., Torr, P.H., Hospedales. T.M.: Learning to compare: Relation network for few-shot learning. In: Proceedings of the IEGE conference on computer vision and pattem recognition. pp. 1199-1206 (2018)² 2
[39] Thatipelli, A.. Narayan, S., Khan. S., Anwer, R.M., Khan, F.S., Ghanem, B.: Spatio-temporal relation modeling for shot action recognition. arXiv preprint arXiv:2 12.05132 (2021) 2
11

---

[40] Tian, Y.. Wang. Y., Krishnan, D., Tenenbaum. J.B., Isola, P.: Rethinking few-shot image classification: good embedding is all you need? In: European Conference Computer Vision. pp. 266-282. Springer (2020)
[41] Tran, D., Bourdev, L., Fergus, R., Torresani, L., Paluri, M.: Learning spatiotemporal features with 3d convolutional networks. In: Proceedings of the IEEE international conference on computer vision. pp.
4489-444 4497 (2015) 2
[42] Vilalta, R., Drissi, Y: A perspective view and survey of meta-lcaming. Artificial intelligence review 18(2),
77-95 (2002) 2
[43] Vinyals, O., Blundell, C., Lillicrap, T., Wierstra, D., et al.: Matching networks for one shot learning. Advances in neural information processing systems 29 (2016)>
[44] Xie, J., Su, X., You, S., Ma, 7., Wang, F., Qian, C.: Scalenet: Searching for the model scale. arXiv preprint vrXiv:227.002666 (2022) 3
[45] Xu, H., Su, X., Wang. Y., Cai, H., Cui, K., Chen, X.: Automatic bridge crack detection using convolutional neural network. Applied Sciences 9(4)) 2867 (2019) 1
[46] Xu, H., Su, X., You, S., Huang. T., Wang, F., Qian, C., Zhang, C., Xu, C.,, Wang, D., Sowmya, A.: Data agnostic filter gating for efficient deep networks. In: ICASSP 2022-2022 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP). pp. 3503 3507. IEEE (2022)
[47] Zhang, H., Zhang. L., Qi, X., Li, H., Torr, P.H., Koniusz, P.: Few-shot action recognition with permutation- invariant attention. In: European Conference on Computer Vision. pp. 525-542. Springer (2020) 3,
[48] Zheng, M.. Wang, F. You, S., Qian, C., Zhang, C., Wang. X., Xu, C.: Weakly supervised contrastive learning. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). pp. 10042 10051 (October 2021) 1
[49] Zheng, M., You, S., Huang, I.., Wang, F., Qian, C., Xu, C.: Simmatch: Semi-supervised learning with similarity matching. In: Proceedings of the IREE/CVF Conference on Computer Vision and Pattem Recognition. pp. 14471-14481 (2022) 1
[50] Zheng, M., You. S., Wang, F., Qian, C., Zhang, C., Wang, X., Xu, C.: Ressl: Relational self- supervised learning with weak augmentation. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang. P., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems. vol. 34, pp. 2543 2555. Curran Associates, Inc. (2021), http::///ppcceedinn neurips. wttpcccpp///////////epcp Paperr-aprrpppppp
[51] Zhou, B.. Andonian. A., Oliva, A.. Torralba, A.: Temporal relational reasoning in videos. In: Proceedings of the European conference on computer vision (ECCV). pp. 803-818 (2018) 2
[52] Zhu, L., Yang. Y.: Compound memory networks for few-shot vidco classification. In: Proccedings of the Europcan Conference on Computer Vision (ECCV). pp.
(2018) 1,2
[53] Zhu, X.. Toisoul, A., Perez-Rua. J.M., Zhang, Martinez, B., Xiang. T.: Few-shot action recognition with prototype-centered attentive learning. arxiv preprint arXiv:2101 100028000000885 (2021) 2
[54] Zhu, Li, X., Liu, C., Zolfaghari, M., Xiong, Y., Wu, C., Zhang. Z., Tighe, J., Manmatha, R., Li, M.: A comprehensive study of deep video action recognition. arXiv preprint arXiv:2012.06567 (2020) 1,2
[55] Zhu, Z., Wang, L., Guo, S., Wu, G.: A closer look at few-shot vidco classification: A new baseline and benchmark. arXiv preprint arXiv:2 10.
1..::2110.123: 12358 (2021) 2,
[56] Zolfaghari, M., Singh, K., Brox, T.: Eco: Efficient convolutional network for online video understanding.
In: Proceedings of the European conference on computer vision (ECCV). pp. 695-712 (2018)
12

---

Checklist
The checklist follows the references. Please read the checklist guidelines carefully for information on how to answer these questions. For each question, change the default [TODO] to [Yes], [No], , or [N/A] - You are strongly encouraged to include justification your answer, either by referencing the appropriate section of your paper or providing a brief inline description. For example:
Did you include the license to the code and datasets? [Yes] See Section ??.
Did you include the license the code and datasets? [No] The code and the data are proprietary.
Did you include the license to the code and datasets? [N/A]
Please do modify the questions and only use the provided macros for your answers. Note that the Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below,
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [No]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as URL)? [N/A]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [No]
(c) Did you include any new assets either the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "vdveXOdNY5", "reviewer": "Reviewer_bQCA", "summary": "This paper proposes a spatio-temporal feature matching and alignment method for the task of few-shot action recognition with the end-to-end design of models. To achieve this aim, the authors construct a transformer space with spatial and temporal attention elements and search for the optimal Spatio-Temporal representations from few-shot videos from the architecture perspective. In detail, this paper introduces a transformer space shrinking strategy to evolve transformer space and speed up the architecture search adaptively. Besides, this paper proposes a more efficient and general spatio-temporal prototype alignment method, which can be conveniently adapted to arbitrary length video matching. This paper is novel, and the promising results compared to baseline methods prove the efficientness of the proposed method.", "questions": "As illustrated in the main review, I curious more about the code, computation budget, and some ablation studies of this algorithm.\n", "limitations": "Yes, the authors adequately discussed the limitations of this paper.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "+ The formulation of transformer space shrinking is good. It is interesting and reasonable that this paper leverages the expectation of subnets and budgets for evaluating different operations. However, this method is a little complicated. Do authors have any plan to open source the code for the contribution of the community?\n \n+ The authors propose a spatio-temporal prototype alignment method. I am considering the efficiency of this new method. Does the proposed method have the same computation budget as the old way?\n \n+ The visualization of searched architectures is very impressive, and the search architectures' analysis seems promising.\n \n+ With much smaller FLOPs and parameters, this algorithm achieves relatively good performance compared to the baseline methods.\n \n- This paper only provides the experiments without pre-trained weights. I am curious about the results of the pre-trained weights with the ImageNet dataset. With this setting, can this algorithm still achieve a promising result?\n \n- Fig 4 and Fig 5 analyze the effect of search space shrinking from the aspect of supernet. I notice that the test loss in Fig 5 starts from 2.5 epochs. Why not show the results from the 0 epoch? Besides, I do not think there is a direct correspondence between the training, testing, and removing useless operations. The supernet may get better training because of the smaller search space rather than removing redundant ops.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "3JumZEX0Pk", "reviewer": "Reviewer_kc2E", "summary": "This paper presents a video model for few-shot action recognition. It has two main contributions: (1) NAS to search for model components from a Transformer search space w/ divided space-time attention (space: selecting Spatial/Temporal attention blocks, selecting attention heads in {6,12,16}), and (2) a Prototype Alignment module to compute similarity scores between query and support sets (similar to TRX [25]). Transformer search space is reduced during training based on a shrinkage strategy. The corresponding shrinkage score has the following intuition to determine the strength of an operator (i.e., S/T attention or different number of heads): If the expected loss of a subnet with a certain operator deviates (beyond a threshold) from the expected loss of subnets with the same compute budget, then the corresponding operator is dropped from the search space. Simply put, an operator is useless if it performs worse at the same budget. The authors validate their method with few-shot settings in UCF101 and HMDB51, but mainly consider models initialized from scratch (w/o any ImageNet pretraining) in contrast to previous work. Ablation studies further provide insights into different components of the proposed design.", "questions": "- Why is this the best search space? This is not well-motivated. I believe the authors should discuss why these dimensions (spatial/temporal/heads) are the ones to search in and whether it is sufficient. There are many other aspects such as spatio-temporal resolution, channel-expansion, parallel/multi-path connections, components beyond self-attention. I agree that including all these axes will be practically impossible, but the choices of authors should be well-motivated compared to other options.\n\n- Is the inequality in Eq. 5 correct? I think one should be reversed. Is it not true that when the budget is higher the loss is lower in general?\n\n- I do not fully understand the definition of a subnet based on its operators (in L125). I see the motivation, but the notation seems off. Why use summation for each operator within a layer but union across layers? Also, what do you mean by saying the sum of all indicator functions in a layer equals to 1? Please clarify this and better represent it in the paper.\n\n[Post-rebuttal]\n- Authors have clarified my questions and I am  satisfied with their answers. ", "limitations": "No major negative societal impact.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- This paper presents a well-motivated idea and sound arguments in general. The selection of spatial/temporal components makes the design simple, and the search more convenient. It makes sense to vary the number of temporal components in settings w/ different number of input frames.\n\n- The claims of the paper are thoroughly validated in the experiments. Also, the provided ablations give insights into each component in the proposed design.\n\n\nWeaknesses:\n\n- The pretraining setup that the authors follow seems to be unfair. In fact, the authors initialize the models from scratch rather than ImageNet-pretrained weights (in contrast to previous work), citing that there is a possible information leak from the pretraining data to few-shot classes. I do agree with this comment. However, the baseline methods that the authors compare against have significantly-larger budget (params/FLOPs) and they may be at a disadvantage compared to the proposed method, if not pretrained on a large-enough dataset. Because, large models tend to overfit more on small training datasets. It would be interesting to see the performance w/o any pretraining at a similar compute budget. In Table 7, authors pretrain on UCF101 which is still a smaller dataset.\n\n- The proposed Prototype Alignment module is well-motivated and a main contribution of the paper. However, I wonder if this is truely novel. It has similarities with the proposal in TRX which also presents a query-specific class prototype for computing distance. Also, the authors call this module 'spatio-temporal alignment' (Section 3.4), when in fact, there is no spatial information at the input to the module. This only performs temporal alignment.\n\n- Some information missing regarding the shrinkage strategy (can be addressed with another equation). It says that an operator is discarded if its score 'deviates' from the expected score at a similar budget. The score could be significantly lower/higher in both directions beyond the threshold, and a lower score should be better as far as I can see. Is it not the case? This should be clarified.\n\n- Writing in the paper is sometimes missing critical information and hard to follow. It can be improved. Make sure the paper is self-contained (eg: L113: 'as in [15]').\n\n- Specific hyperparameter settings are missing, especially in the shrinkage algorithm, which is probably important to include/discuss.\n\n\n[Post-rebuttal]\n- Authors have responded to all my concerns and I am  satisfied with their answers. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "2l4RgRUa9e5", "reviewer": "Reviewer_4Tcp", "summary": "This paper presents a neural architecture search method for few-shot action recognition. The main contribution is the proposal of transformer space shrinking strategy and spatio-temporal prototype alignment. Experiments are conducted on HMDB51 and UCF101.", "questions": "1. Performance comparison with the state-of-the-art methods when using pre-trained weights\n2. Experimental results on large-scale datasets", "limitations": "See above", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. Ablation studies are conducted to evaluate the effectiveness of each component.\n2. The proposed transformer space shrinking strategy reduces the time and cost for architecture search.\nWeaknesses:\n1. I understand that there may be some contradictions between pre-training and few-shot learning. However, since almost all of the state-of-the-art methods are based on pre-trained weights instead of random initialization, the authors should also report performances with pre-trained weights to ensure comparability among them.\n2. To demonstrate the superiority and generality of the proposed method, experimental evaluations should be performed on large-scale datasets of action recognition, such as Kinetics and Something-Something, just like the competitors did.\n3. Some closely related works, such as [a, b], should also be introduced and compared in the paper.\n[a] Xiatian Zhu, Antoine Toisoul, Juan-Manuel Perez-Rua, Li Zhang, Brais Martinez, and Tao Xiang. \"Few-shot action recognition with prototype-centered attentive learning.\" In BMVC, 2021.\n[b] Shuyuan Li, Huabin Liu, Rui Qian, Yuxi Li, John See, Mengjuan Fei, Xiaoyuan Yu, and Weiyao Lin. \"TA2N: Two-Stage Action Alignment Network for Few-Shot Action Recognition.\" In AAAI, 2022.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept