Maximizing Spatio-Temporal Entropy of Deep 3D CNNs for Efficient Video Recognition
Authors: Junyan Wang
Zhenhong Sun
Yichen Qian
Dong Gong
Xiuyu Sun
Ming Lin
Maurice Pagnucco
Yang Song
Venue: ICLR 2023 poster (2023)

Abstract: 3D convolution neural networks (CNNs) have been the prevailing option for video recognition. To capture the temporal information, 3D convolutions are computed along the sequences, leading to cubically growing and expensive computations. To reduce the computational cost, previous methods resort to manually designed 3D/2D CNN structures with approximations or automatic search, which sacrifice the modeling ability or make training time-consuming. In this work, we propose to automatically design efficient 3D CNN architectures via a novel training-free neural architecture search approach tailored for 3D CNNs considering the model complexity. To measure the expressiveness of 3D CNNs efficiently, we formulate a 3D CNN as an information system and derive an analytic entropy score, based on the Maximum Entropy Principle. Specifically, we propose a spatio-temporal entropy score (STEntr-Score) with a refinement factor to handle the discrepancy of visual information in spatial and temporal dimensions, through dynamically leveraging the correlation between the feature map size and kernel size depth-wisely. Highly efficient and expressive 3D CNN architectures, i.e., entropy-based 3D CNNs (E3D family),  can then be efficiently searched by maximizing the STEntr-Score under a given computational budget, via an evolutionary algorithm without training the network parameters. Extensive experiments on Something-Something V1&V2 and Kinetics400 demonstrate that the E3D family achieves state-of-the-art performance with higher computational efficiency.

Published as a conference paper at ICLR 2023
MAXIMIZING
ENTROPY OF DEEP 3D CNNS FOR EFFICIENT VIDEO RECOGNITION
Junyan Wang Zhenhong Sun 128 Yichen Qian', Dong Gong', Xiuyu ;un?;Ming Lin
Maurice Pagnucco', Yang Song'
l University of New South Wales
DAMO Academy, Alibaba Group 3 AAmaoon
(junyan. wang, dong.gong, sang.song]] edu au
(zeehhogg szh, yichen. gyc,, xiuyu (ilyy.sxy)aa sxy) @a babaaiinee com (morri) (morri]ecse. Jcse. unsw, edu.au (minglamz) 3 ccom
ABSTRACT
3D convolution neural networks (CNNs) have been the prevailing option for video recognition. To capture the temporal information, 3D convolutions are com- puted along the sequences, leading to cubically growing and expensive compu- tations. To reduce the computational cost, previous methods resort to manually designed 3D/2D CNN structures with approximations automatic search, which sacrifice the modeling ability or make training time-conuuming In this work, we propose automatically design efficient 3D CNN architectures via a novel training-free neural architecture search approach tailored for 3D CNNs consid- ering the model complexity. To measure the expressiveness of 3D CNNs effi- ciently, we formulate a 3D CNN as an information system and derive an ana- lytic entropy score, based on the Maximum Entropy Principle. Specifically, we propose a spatio-temporal entropy score (STEntr-Score) with a refinement factor handle the discrepancy of visual information in spatial and temporal dimen- sions, through dynamically leveraging the correlation between the feature map size and kernel size depth-wisely. Highly efficient and expressive 3D CNN ar- chitectures, i.e., entropy- -basep 3D CNNs (E3D family), then be efficiently searched by maximizing the STEntr-Score under a given computational budget, via evolutionary algorithm without training the network parameters. Exten- sive experiments on Something-Somethinn V1& V2 and Kinetics400 demonstrate that the E3D family achieves state-of-the-art performance with higher compu- tational efficiency. Code is available at tts: com/ allibbaa// lightweight neural- architecture search
INTRODUCTION
Video recognition a fundamental task for video understanding. To capture the visual information in both temporal and spatial domains from high-quality large-scale videos, most works have been focusing on proposing highly expressive models which, however, lead to higher computational costs Kondratyuk al. (2021) : Zhang et al. (2022): Li al. Recent research shows that 3D CNNs achieve excellent performance on large-scale benchmarks (Hara et al.) 2018) with unified computations to capture spatio-temporal features jointly. However, the computational cost grows cubically in stan- dard 3D convolution, making it prohibitive for high-resolution long-duration videos. Previous works propose to improve the efficiency of 3D CNNs via 2D decomposition or approximation manually (Carreira & Zisserman 2017 Tran et al. 2018 Feichtenhofer 2020). Some practices have also been conducted to manually design efficintt 3D CNNs relying on heuristics or experiences (Hara et al.] 2018 Feichtenhofer (202)) The manually designed 3D or 2D CNN structures cost massive ef- forts and time strengthening the modeling ability. Neural Architecture Search (NAS) approaches (Kondratykk et al. 2021 Wang et al. 2020) can automatically generate 3D CNN architectures with
"Corresponding author, Pequal contribution, wark done Alibaba.

---

Published as a conference paper at ICLR 2023
higher modeling ability. However, searching for a single 3D architecture requires days on multi- ple GPUs or TPUs, as training and evaluation of the accuracy indicator are required in the process, making the automatic 3D CNN design process time-consuming and/or hurdware-dependers
To tackle the above issues, we study how to automatically generate (or design) efficient and expres- sive 3D CNNs with limited computations. Recently, training-free -ree technologies have been introduced by some approaches (Chen et al., [021]] Lin et al.; [02T]] [Snn et al.] 2022b). in which kernel spec- trum analysis or forward inference are adopted measure the expressiveness of spatial 2D CNNs. Inspired by the training-free frree concept and information theory, we suggest that deep network can be regarded as an information system, and measuring the expressiveness of the network can be considered equivalent to analyzing how much information it can capture. Therefore, based on the Maximum Entropy Principle [Jaynes] 1957). , the probability distribution of the system that best represents the current state of knowledge is the one with the highest entropy. However, as discussed in (Xie et al. 2018), the information in spatial and temporal domains is different in natural video data. The spatial dimension is usually limited to some local properties, like connectivity (Claramunt) 2012). while the temporal dimension usually contains more drastic variations with more complex information. To address the spatio-tem temporal discrepancy in video data, we conduct a kernel selection experiment and observe that different 3D kernel selections in different stages have different effects on performance, and the focus of 3D CNNs changes from spatial information to spatio-temporal in- formation, as the network depth increases. We thus consider that the design of 3D CNN architecture should focus on spatial-temporal aggregation depth-wisely.
The above analysis has motivated us to propose a training-free NAS approach to obtain optimal architectures, i.e., entropy-based 3D CNNs (E3D family). Coneretely, we first formulate 3D CNN-based architecture as an information system whose expressiveness can be measured by the value of its differential entropy. We then derive the upper bound of the differential entropy using an analytic formulation, named OppatioTTeporra Entropy Score (STEnt-- Sconee,,,,,,,, conditioned on spatio-temporal aggregation by dynamically measuring the correlation between feature map size and kemnel size depth-wisely. Finally, an evolutionary algorithm is employed to identify the optimal architecture utilizing the STEntr- Score without training network parameters during searching. In summary, the key contributions of our work are as follows:
We present a novel training-free neural architecture search approach to design efficient 3D CNN architectures. Instead of using forward inference estimation, calculate the differential entropy of a 3D CNN by an analytic formulation under Maximum Entropy Principle.
We investigate the video data characteristics in spatial and temporal domains and correlation be- tween feature map with kernel selection, then propose the corresponding spatio-temporal entropy score estimate the spatio-temporal aggregation dynamically, with spatio-temppral refinement mechanism to handle the information discrepancy.
Each model of E3D family can be searched within three hours on a desktop CPU, and the models demonstrate state-of- of-the-art performance on various video recognition datasets.
2 RELATED WORK
Action recognition. 2D CNNs lack temporal modeling for video sequences, and many approaches
(Wang et al. 2016 Lin et al. 2019 ei al. 2020: Wang et al. 2021a b Huang et al. 2022) focused
on designing an extended module for temporal information learning. Meanwhile, 3D CNseb frameworks have a spatio-temporal modeling capability, which improves model performance for video action recognition (Tran et al. 2015 Carreira & Zisserman Feichtehie,,,,ofeer, 2020 Kon-
dratyuk et al. 2021). Some attempts (Feichtenhofer] |020 Fan et al. [020 Kondratyuk et al. 2021) focused on designing efficient 3D CNN-based architectures. For example, X3D (Feichtenhofer)
2020) progressively expands tiny 2D image classification architecture along multiple network axes, in space, time, width and depth. Our work also focuses on designing efficient 3D CNN-based architectures, but in deterministic manner with entropy- based information criterion analysis.
Maximum Entropy Principle. The Principle of Maximum Entropy is one of the fundamental prin- ciples in Physics and Information Theory (Shannon 1948: Reza 1994; Kullback 1997; Brillouin 2013). Accompanied by the widespread applications of deep learning. many theoretical studies
Saax et al. 2019 Chan et al. 2021 Yu et al. 2020 Sun et al. 2022b) try to understand the success
2

---

Published as a conference paper at ICLR 2023
of deep learning based on the Maximum Entropy Principle. Our work focuses video recognition and explores the aggregation of -emporll information under the Maximum Entropy Principle.
Training-Free NAS. To reduce the search time of NAS, recent attempts (Mellor al. 202 Chen
ct al.
2021] Tanaka et al. 2020: Lin ct al. 2021; Sun et al. 2022a b Zhou et al., 2022
Chen et al.
2022
Lin al., 2020 proposed training- free strategies for architecture searching, which construct an alternaiiee score to rank the initialized networks without training. For example, the work of (Sun)
al., 2022b) maximizes the differential entropy of detection backbones, leading to a better fea- ture extractor for object detection under the given computational budgets. However, these methods construct scores on spatial 2D CNNs, and cannot handle the discrepancy of visual information in spatial and temporal dimensions of 3D CNNs. In order to address the above issues, our work aims to optimize the network architecture by considering spatio-temporal dimensions aggregation.
THE PROPOSED APPROACH
In this section, first present detailed technical description of the derivation process of an alytical solution and propose the STEntr-Score with a refinement factor to handle the discrepancy of visual information in spatial and temporal dimensions. Then we give overview of the search strategy for the E3D family, via an evolutionary algorithm without training the network parameters.
3.1 PRELIMINARY
In Information Theory, differential entropy is employed to represent the information capacity of an information system by measuring the output of the system (Jaynes 1957 Reza 1994; Kullback, 1997 Brillouin| 2013 Norwich 1993). Generally, the output of system a high- dimensional con- tinuous variable with complex probability distribution, making it difficult to compute the precise value of its entropy directly. Based on the Maximum Entropy Principle (Jaynes 1957), ). common alternative approach is to estimate the upper bound of the entropy (Cover & Thomas 2012), as:
Theorem For any continuous distribution P(x) of mean μ and variance o², its differential en- tropy is maximized when P(x) is Gaussian distribution (p,o²)
Thus, the differential entropy of any distribution is upper bound by the Gaussian distribution with the same mean and variance. Suppose æ is sampled from Gaussian distribution N N(,, o²), the dif- ferential entropy (Norwich) 1993) of is then:
H(x) = - ((x) x log(o?);
where P(x)) represents the probability density function of x. Note that the entropy of the Gaussian distribution depends only on the variance o², and simple proof is included in the Appendix B.1
According to successful deep learning applications (Saxe et al., 2019: Chan et al. 2021 Yu et al. 2020 Sun et al. 2022b a of Maximum Entropy Principle, deep neural network can be regarded as information system, and the differential entropy of the last output feature map represents the expressiveness of the system. Recent method (Sun et al. (022b)) estimates the entropy of 2D CNNs by simply computing the feature map variance via sampling input data and initializing network parameters from a random standard Gaussian distribution. However, when migrating to 3D CNNs, how to efficiently reduce the random sampling noise due the random initialization, and how to estimate the entropy after aggregating spatial and temporal dimensions in 3D CNNs design, still remain open questions. We then propose our method address these problems.
3.2 STATISTICAL ANALYSIS OF ENTROPY IN DEEP 3D CNNS
Simple Network Space. Following the idea that simpler is better (Lin et al. 2021 1- Sun et al. 2022bb)
we apply vanilla 3D CNNs without considering auxiliary modules (e.g., BN (loffe & Szegedy 2013), Reslink (He et al., (016)) SE block (Hu et al., 2018) and so on) to conduct analysis of network architectures. Formally, given a convolutional network with L layers of weights W! 4 the forward inference with a simple network space is given by:
x² W x
for I 1, 1,.... Te
3

---

Published as a conference paper at ICLR 2023
where x' denotes the 1th layer feature map. For holistic analysis, the bias of the convolutional layer is set to zero and the activation function omitted in the network for simplification. Auxiliary modules ignored during entropy calculation and plugged into the backbone without special modification during training. A detailed discussion about these rules is included in Appendix C
Since the imput data and network parameters randomly sampled from Gaussian distributions, the forward entropy calculation will be inconsistent, which might lead to random sampling noise. To obtain valid entropy value, computing an average value from multiple sampling iterations and increasing the value of batch size or resolution can be adopted to reduce the noise. These operations are however consuming and cost higher computational resources. To this end, we propose to explore the statistical characteristies of the forward inference, to provide an efficient solution.
Maximum Entropy of 3D CNNs. We first consider the product law of expectation (Mood 1950) and the Bienaymé's 's identity in probability theory (Loeve 2017), as follows:
Theorem 2 Given two independent random variables tn U2 the expectation of their product 0102 is: E(U1 122) ((v)))((v))
Theorem Given n random variables {U1, 02, Us, u..,..., Un which are pairwise independent integrable, the sums of their expectations and variances B( vi) >i E(vs), and (v()).
D'
21) D
We can thus compute the expectation and variance of 1th layer feature map element x as:
K
K
Gl-1
B(x²)=
Bl ((Wcthhu))))
B(x 1-1
(3)
R
K
c
D²(2)) =
1 (x as (D (WW
E(W
(4)
D ( Wathow [((c'rhw))
where (K), K/ Ky represents the kernel size of the 7th layer in the 3D CNN, and Ct denotes its input channels size. Note that Ch-1 is equal to 1 when the layer is depth-wise convolution. Besides, f, ft, denote the temporal, height, and width positions, respectively. A simple proof is included Appendix B.2
The input x: is initialized from a standard Gaussian distribution, which means that its expectation B(=²)= and variance (2:)) 1. From the perspective of statistics, when sampling sufficient times. Also, suppose that all parameters are initialized from zero-mean Gaussian distribution, and thus the variance of the last layer D2 can be computed by propagating the variances from previous layers as:
regard DD2 (=
(7
II
(5)
Finally, by combining Eq. 5 and Eq. proportional to:
we derive that the upper bound entropy is numerically
L log Lo - (W
H(F) X
(6)
where detailed proof is included in Appendix B.3 By assuming that the parameters of each layer are initialized with a standard Gaussian distribution with D(W! (itto) 1, the entropy score defined in Eq. (6) be written El_ log( KK K C² It measures the influence of kemnll size and channel dimension on the entropy score in a homogeneous way, named HomoEntr-Score. This analytic formulation does not require random sampling, thus random sampling noise exists.
4

---

Published as a conference paper at ICLR 2023
3.3 SPATIO-TEMPORAL ENTROPY SCORE
The Semnonntt-Soorr is derived from the analysis with an independent and identical assumption on the imput elements (and the corresponding intermediate features). Although can generally repre- sent the expressiveness characteristics of neural network, there is a gap between HomoFntr-Scomm and reality on 3D CNNs. When directly applying it on 3D CNNs for handling video sequences, we realize the HomoEntr-Score with the independent and identical assumption cannot capture the dis- crepancy of the visual information in the spatial and temporal domain, as the information between spatial and temporal dimensions in video data is different in video recognition. The gap leads to some issues with HomoEntr-Scone for modeling video data with 3D CNNs. The observations will be discussed and analyzed in the following. Note that HomoEntr-Sear Scort (and similar approaches (Sun et al., 2022b)) can work well for modeling the expressiveness of 2D CNNs since there is no (obvious) discrepancy on the information of the two directions in 2D images statistically. Based on the analyses, we propose Temporal Entropy Score (STEntr-Score for 3D CNNs on video data, where a Spatio-temporal refinement factor is introduced to handle the information discrepancy.
Params (M)
FLOPs
HomoEntr Score
Params (M)
FLOPs (G)
HomoEntr Score
Model
Tup1
Model
Top1
(G)
S-2
45.15%
3.33
1.93
178.5
T-2
41.59%
3.32
1.93
177.7
S-3
44.87%
3.33
1.93
178.4
T-3
42.93%
3.32
1.93
177.8
S-4
44.35%
3.33
1.94
178.4
T-4
43.17%
3.32
1.92
177.8
S-5
43.85%
3.33
1.94
178.4
T-5
43.43%
3.32
1.92
177.9
S-6
43.83%
3.33
1.94
178.4
T-6
43.35%
3.32
1.92
177.9
Table I: Results of different kernel positions on the Sth-Sth VI validation dataset. All model structures are based on X3D-S (Feichtenhofer (020)). "S-N" models mean only stage N selects 1x5x5 kernel, and others select 3x3x3. "T-N" models mean only stage N selects 3 3x3x3 kernel, and others select 1x5x5. Note that we divide stage 4 of X3D with 11 layers into two stages (5 and 6 layers).
Kernel Selection Observations. We conduct an experiment to explore how different 3D convolu- tional kernel sizes at different stages (i.e., layer blocks at different positions in the network) impact the performance, shown in Table I1 All models are based on X3D-S but with different kernels in different stages. We set 1 xx55 and 3x3x3 kernels at the different stages in the 3D CNNs, which typical 3D convolutional kernels for learning spatio-temporal information. These two different choices enable a layer to aggregate the visual information focusing on different spatial and temporal dimensions, with the receptive field of CNN in the most pertinent directions. In Table the perfor- mances of $-2 and S-3 models are higher than X3D-S with only 3x3x3 kernels (44.6% in Table(2). and S-series outperform T-series, which show that kernel selection at different stages influences the performance significantly, and that different stages may prefer different kernel sizes, respectively. Although the kernel selections (with different spatio-tempors dimensions) at different stages lead to different effects on performance, the corresponding 3D CNNs have similar HomoEntr-
Resplution
Stages
Stages
Stages
Stages
Figure 1: Input feature map size and kernel sizes of S-2, T-5 and X3D model in cach stage.
According to the downsampling strategy of the 3D CNNs, spatial resolutions become smaller from the large input as the depth increases, while the temporal frame size remains a certain value, as shown in Figure Db. Through analyzing the results in Table 1 we can infer that spatial kernels (like 1 1x5x5) can obtain spatial information more effectively at low-level stages, and apatio-tempora kernels (like 3x3x3) are more stable to obtain ppttiottemporal information at high-level stages. Meanwhile, the similarity between the input feature map and the kernel size of the S-2 model at each stage is higher than that of the T-5 model or X3D, according to Figure T We thus consider that
5

---

Published as a conference paper at ICLR 2023
with the higher correlation between the feature map size and kernel size depth-wisely. the model can obtain higher expressiveness spatial and temporal information.
Temporall Refinement. To estimate the correlation between feature map and kernel size in different depths, we first define two vectors: the input feature map size = [I, H. , W] and the 3D kernel size K [K Kn, Kw, in convolutional layer, where {T, H, W) I represent frame, height and width dimension size. We compute the distance D based on commonly used cosine distance as:
D(S, K) = -logg (eoosine (S, K)) -log( S K |||] K
where Deosine represents the Cosine Distance function, and we expand the diversity of the cosine distance by using log. We thus utilize the distance D between S and K in each layer define the variance of weight dynamically. Finally, we refine the upper bound differential entropy as:
H(F) &
log log( KK . D(S),K))
We name this analytic formulation of Eq. as Spatio-Temporal Entropy Score (STEntr-Score) to measure the aggregation of spatio-temporal dimen- sions. After spatio-temporal refinement, calculate the entropy of each model by STEntr-Score in Table [ and present the relationship between ac- curacy with STEntr-Score and HomoEntr-Score in Figure 2 According to this figure, STEntr-Score is positively correlated with Topl accuracy which in- dicates that the proposed spatio-temporal refinement can handle the discrepancy of visual information in spatial and temporal dimensions.
46 Samoer
STEnEnScore
TW 0oonsrrt-oooa (evestet seaee
1735
1850
185.0
195.0 197.5
Enerpy Score
Figure 2: Top-1 accuracy STEnte-Score and HomoEntr-Scomp
3.4 3D CNN SEARCHING STRATEGY
Utilizing STEntr-Score, apply the basic Evolutionary Algorithm (EA) find the optimal 3D CNN architectures, which is similar to (Lin et al. 22021] Sun et al. 2022b). We initialize a population of candidates randomly under small budget and detine the 3D kernel search space within each layer with two options: {1 (I-aaace)2, ftimes x (k-ppoee2)), then randomly select two stages from the candidates and mutate them each iteration step. We calculate its STEntr- the evolution process instead of evaluating the accuracy after mutation, if the inference cost of the mutated structure does not exceed the budget. The population will be maintained to a certain size during iterations, by discarding the worst candidate of the smallest STEntr-Score. After all iterations, the target network is achieved with the largest STEntr- SSentr--correae under the given budget (e.g., FLOPs, parameters, and latency). Since the latency budget requires forward process GPU which will diminish the efficiency of our STEntr-Score search, we choose FLOPs as the target budget. Another reason for applying FLOPs budget is to fairly compare with X3D (Feichtenhofer 2020) and Mo ViNet (Kondratyuk et al.) 2021). which only report FLOPs rather than latency. Thus, we obtain the spatio-temporal entropy 3DD CNNs family (E3D family) under certain FLOPs, All models are searched separately with different FLOPs bugdet (1.9G, 4.7G, and 18.4G) for fair comparison
to navigate
with X3D-S/M/L as the baseline, and the detailed algorithm described in Appendix E
4 EXPERIMENTS
Our E3D family consists of E3D-S (1.9G FLOPs), E3D-M (4.7G FLOPs), and E3D-L (18.3G FLOPs). The detailed structures of the E3D family deseribed in Appendix F We compare approach with other state-of of-te-of-the-art methods and in-depth analysis to better understand our method. More results presented in Appendix

---

Published as a conference paper at ICLR 2023
Method
Backbone
Pretrain
Resolution
GFLOPs
1x1 VI-Val
2x3 V2-Val
Top1
Top5
Top1
Top5
ISN (Wang
2016
ResNet50
ImageNet
256'
16
TSM Lin al 2019
ResNet50
ImageNet
16 x 256
65
47.2
77.1
63.4
88.5
TANel [1 ng all
Ar21
ResNet50
ImageNet
16 x 2562
66
47.6
77.7
64.6
89.5
ActionNel
2021b
ResNet50
ImageNet
x 256
69.5
89.3
TAda (Huang al. 2022
ConvNeXi-T
ImageNet
16 256
47
64.8
88.8
B3D
(Careiia & Zisserman 2017
Inception
InageNet+Ka00
64 x 256
306
41.6
72.2
NL
(Cariera & Zisserman 2017
InceptionV1
ImageNet+K400
64 x
334
44,4
76.0
S3D-G
et
Inception
ImageNet
64 256
71,4
48.2
78.7
X3D
Fetcheebboler
2020
X3D-S
No pretrain
13 x 1602
2
44.6
74.4
60.1
85.9
X3D+
Fotchtenholer
2020
X3D-M
No pretrain
16 x 224
4.7
47.3
76.6
62.2
87.2
X3D
Feichtenhoter
2020
X3D-L
No pretrain
16 3122
1844
49.4
77.9
MoViNeTe
[Knndralynk cl 202
MoViNet-AO
No pretrain
50 x 1722
2.7
46.9
75.0
61.9
87.2
MoViNete
(Kondralyuk
2021
MoViNet-A1
No pretrain
50 x 1722
6
49.3
77.1
64.5
89.1
E3D-S
pretrain
160"
1.9
47.1
75.6
87.6
E3D
E3D-M
No pretrain
16 224
4.7
49.4
78.1
64.7
89.6
E3D
E3D-L
No pretrain
16 X 312
18.3
51.1
78.7
65.7
89.8
Table 2: Comparison with state-of the-art methods on Sth-Sth VI and V2 validation datasets. The models only take RGB frames imputs. To be consistent with compared approaches, present most results of 2D CNN- based methods with ResNet50. "MN-V2"" denotes MobileNet-V2. kxk denotes temporal clip with spatial crop evaluation. "." indicates the results are not available for us, and denotes our reproduced models.
4.1 EXPERIMENT SETTINGS
The E3D family includes search stage without training and training & inference stage for video recognition on a specific dataset. Detailed settings in each stage are described as follows:
Search Settings. Following X3D (Feichtenhofer 2020), we also apply MobileNet-like network basis, in which the core concept is 3D depth-wise separable convolution for efficiency. The initial structure composed of 5 stages with small and narrow blocks to meet the reasonable budget. which is usually below 1/3 of the target FLOPs budget. The population size and total iterations of EA are set as 512 and 500000, respectively. During mutation stages from the candidates, we randomly select 3D kernels from x1x3x3, x,, 3 3x3x3) to replace the current one; interchange the expansion ratio bottleneck from {1.5, 2.0, 2.5, 3.0, 3.5, (botttllle = ratio x intput); scale the output channels with the ratios {2.0, 1.5, 1.25, 0.8, 0.6, 0.5}; increases or decreases depth with or 2. Note that the channel dimension of every layer is fixed within 8 to 640 with multiples of 8, which helps shrink homologous search space and accelerate search speed.
Training & Inference. Our experiments are conducted on three large- datasets, Something- Something (Sth-Sth) V1&V2 (Goyyll et al. (017)) and Kinetics400 (Kay et al., (017). All models trained by using Stochastic Gradient Descent (SGD). The cosine learning rate schedule (Loshchilov & Hutter 2016) is employed, and total epochs set to 100 for Sth-Sth V1&V2 datasets, and 150 for Kinetics400 dataset, with synchronized Batch-Norm instead of common Batch-Norm. Random scaling, cropping, and flipping are applied data augmentation on all datasets. To be comparable with previous work and evaluate accuracy and complexity trade-offs, we apply two testing strategies: 1) K-Center: temporally, uniformly sampling of K clips (e.g., K=10) from video and taking a center crop. 2) K- K-LefiCenterRigh: also uniformly sampling K clips temporally, but taking multiple crops to cover the longer spatial axis, as an approximation of fully-convolutional testing. For all methods, we follow prior studies by reporting Topl and Top5 recognition accuracy, and FL.OPs to indicate the model complexity. More experiment setting details can be seen in Appendix G
4.2 MAIN RESULTS
Sth-Sth V1&V2. Tabel 2 shows the comparison between E3D family and state-of-the-art meth- ods. It can be seen that our proposed E3D family achieves competitive performance with more efficient FLOPs-level, which indicates that the E3D models can recognize actions effectively and efficiently. 1) Compared to 2D CNN- methods, E3D outperforms most previous approaches on the same Feols-level. Even compared to many methods with similar performance, requires much lower computational costs. Note that our E3D family does not need to be pretrained on other datasets, and the performance of these 2D CNN-based methods is based ResNet50
model

---

Published as a conference paper at ICLR 2023
Method
Backbone
Pretrain
Frame
Parmm
GFL.OPs Views
Top1
Top5
ISN Wang
2016
RexNet50
ImageNet
25
24.3M
80x1x10
72.5
90.2
TSM
LB 0t M
2019
ReaNe150
ImageNet
16
24.3M
65x3x10
74.7
91.4
TEA
20:0
ReaNet50
ImageNet
16
70x3x10
76.1
92.5
TANe
el alal
2022
RexNet50
ImageNet
16
26M
86x3x
76.9
92.9
IDN
Wanz
202
ReaNet50
ImageNet
16464
71x3x10
77.5
93.2
R(2+
Tain al. 2018
RexNet50
ImageNet
16
63.6M
67x3xx10
73.7
91.6
SlowOnly Feccheniroeer
2019
ReaNet50
ImageNet
8
42x3x10
74.8
91.6
SlowFas Teichienhhler et 2019
RexNei50
ImageNet
8+32
34.4M
x3.7
77.0
92.6
13D
Zivvemnan
InceptionV1
ImageNet
64
108xNIA
Two Two-Stream 130 (Carreira & Zisserman 2017
InceptionVI
TetgeNet
64
25M
216xNIA
75.7
92.0
InceptionV1
ImageNet
X3D Ferentenmeeer 2020
X3D-M
No pretmin
16
3.8M
6,2x3xx10
76.0
92.3
X3D hold MMD
X3D-L
pretrain
248833x10
77.5
92.9
MoViNar (kmndraryxx
2021
MoViNet-A2
No pretmin
50
4.6M
10.3x1x
75.0
92.3
TimeSforne-SS (Bertasius et al. 2021
VTT-
ImageNet
8
121.4M
390x3x10
78.0
93.7
Swin (Liu
Swin-T
ImageNet
32
28.2M
88x3x4
78.8
93.6
E3D
E3D-M
No pretrain
16
3.4M
4.7x3x10
76.4
92.5
E3D-L
pretrain
5.8M
18.3x3x100
77.6
92.9
Table Comparison with state-of -ort---rttttttttte methods on the validation set of Kinetics400. report the inference cost with a single "view" (temporal clip with spatial crop) x the number of such views used (GFLOPsx views). "N/A" and "_" indicate the numbers are not available for us.
stronger backbone that is not suitable for low-level computation. 2) The E3D family also achieves higher performance compared to 3D CNN- methods, which indicates that the architecture of E3D can handle the discrepancy of visual information in spatial and temporal dimensions Compared to the NAS-based method (Kondratyuk et al.) 2021), , our proposed E3D can still achieve a remark- able result which thus verifies the effectiveness of the STEntr-Score for searching the architecture.
Kinetics400. Table shows that E3D achieves sfate-of-the-ar performance compared to most 2D and 3D methods, but uses much less computational resources. Most methods apply ImageNet pretrained backbones on the Kinetics400 dataset. However, our model can still achieve excellent results without using pretrained models, which indicates that our searched architecture by STEntr- Score can effectively learn spatio-temporal information. models (Carreira & Zisserman, 2017 Xie al. 2018 Feichtenhofer 2020) which only employ 3 3x3x3 kernel. It means that kernel se selection is important for action recognition, and STEntr-Scom can benefit 3D CNN architecture design.
2) E3D outperforms other 3D CNN-based
Even though the performance of Transformer-bias models (Bertasius et al. 2021, Neimark et al. 2021 [iu et al. (0222) is competitive, our model still provides remarkable results by using much lower computational resources (FI.Oss) and parameters, which means model is more suitable in efficient scenarios.
4.3 CORRELATION STUDY
Top-1 (%6)
12
40-
S.N.T.
S.N.TT
Fondor
Paroken
Rander
Randein
259
Sre
HomoEntr- Score
FLOPs
Parameters
Figure 3: Correlations between Topl and STEntr-Score, Hooooett-Score, FLOPs, and Parameters. Points represent different sampled models, which have different channel numbers layer configurations.
To verify the importance of STEntr-Score in the design of video understanding models, we ran- domly construct 60 different models (0.2 to GFLOPs) with different channel dimensions and layer numbers investigate the correlations between STEntr- Score, HomoEntr-Scome FLOPs and param- eters. For a fair comparison, all networks are trained on the Sth-Sth VI dataset with batch size of 256 and 50 epochs. We also provide the performance of E3D-S and X3D>-S under the training
8

---

Published as a conference paper at ICLR 2023
setting. According results in Figure more positively correlated with Top1 accuracy than other metrics, which proves the effectiveness of proposed STEntr- Score in evaluating network architecture. (2) Although HomoEntr-Sewwe discriminative on different FLOPs levels, the ability to capture the discrepancy of the visual in- formation in the spatial and temporal domain is not good ST on the same FLOPs level. (3) Benefiting from STEntr-Score, EA can help us obtain 3D CNN architectures with higher expressiveness measured by STEntr-Score on the same FLOPs or parameters level.
observe that: (1) The proposed STEntr-Scone is
4.4
DISCUSSION
Homeent--Scare
Forward 1000 samples
Forward times
16
(ms)
1.5
Score
14
0.5
20
40
60
80
100
120
20
40
60
80
100
120
Input Channels
Input Channels
(a) Consisteney comparison.
(b) Efficiency comparison.
Figure 4: Comparisons between HomoEntr-Seun and "Forward" calculations. "Forward" represents using the forward inference. The calculations conducted AMD Ryzen 5 5600X 6-core CPU.
Comparison with forward inference. For a fair comparison with the realization of forward infer- ence (Sun et al.) 2022b). we use HomoEntr-S -Sorr and conduct a simulation in a three-layer 3D network. The shape of the input feature is 5 5, kernel sizes set to
1, x 3 3 and 1x1x1 with stride of 1, and channels are all set to C.n E (16, 32, 64, 1283. The entropy of each network is caleulated 103 times with either forward inference via Eq. HomoEntr-Score. When performing the forward inference of the network, convolution blocks are re-initialized based Gaussian distribution during each iteration. The filled "Forward" range in Figure 4a demonstrates there exists variance between different random samples, which also empha- sizes the stability of the analytic formulation. Figure 4b] regardless of how channels change, the speed of 10 times formulaic calculation of value remains constant, while the speed reduces almost linearly when performing forward inference. More comparison analysis of training-fres scores included in Appendix
① direct computation of
Searching cost comparison. Since we ap- ply analytic formulation rather than infer- ence, the calculation our STEntr-Score has lower hardware requirements, which means that CPU resources can meet it instead of GPU TPU. From Table 13 our method only takes three hours of searching time with a desktop CPU, while ViNe consumes 24 hours with 64 commercial TPUs. Extremely low time and power consumption demonstrate the searching efficiency of our analytic entropy formulation.
Method
Search
Search
Power Consumption
GFLOPs
TOP-1
MoViNaa-A
TPUs $
24h
69L2KW1
6
49.3
E3D-M
CPU
0.195kWh
4,7
49.4
Table 4: Searching cost comparison on the Sth-Sth VI
dataset. 8 64 Google TPUv3, Power 450W per TPUv3; f: 1 AMD Ryzen 5 5600X 6-Core CPU, Power 65W;
CONCLUSION
In this paper, we propose to automatically design efficient 3D CNN architectures via an entropy- based training-free neural architecture search approach, to address the problem efficient action recognition. In particular, we first formulate the 3D CNN architecture as information system and propose the "TEntr-Score to measure the expressiveness of the system. Then we obtain the E3D family by an evolutionary algorithm, with the help of STEntr-Score. Extensive results show that our searched E3D family achieves higher accuracy and better efficiency compared to many offte-of-the- art action recognition models, within three desktop CPU hours searching.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
This research was supported by Alibaba Group through Alibaba Research Intern Program, and ARC DECRA Fellowship DE230101591 to D. Gong.
REFERENCES
Gedas Bertasius, Heng Wang, and Lorenzo Torresani. Is space-time attention all you need for video understanding? In International Conference on Machine Learning, pp. 813-824. PMLR, 2021.
Leon Brillouin. Science and information theory. Courier Corporation, 2013.
Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? new model and the kinetics dataset. In proceedings of the IEEF Conference on Computer Vision and Pattern Recognition, pp- 6299-6308, 2017.
Kwan Ho Ryan Chan, Yaodong Yu, Chong You, Haozhi Qi, John Wright, and Ma. ReduNet: A white-box deep network from the principle of maximizing rate reduction. arXiv preprint
arXiv:2105.10446 2021.
Hanlin Chen, Ming Lin, Xiuyu Sun, and Hao Li. NAS-bench-zero: A large scale dataset for un- derstanding zero-shot neural architecture search, 2022. URL httpss://peereeviww..t net/
Wuyang Chen, Xinyu Gong, and Zhangyang Wang. Neural architecture search on imagenet in four gpu hours: A theoretically inspired perspective. In Proceedings of the International Conference on Learning Representations, 2021.
Christophe Claramunt. Towards spatio-temporal form of entropy. In International Conference on Conceptual Modeling, pp. 221-230. Springer, 2012.
Thomas M. Cover and Joy A. Thomas. Elements of Information Theory. John Wiley & Sons, 2012.
Linxi Fan, Shyamal Buch, Guanzhi Wang. Ryan Cao, Yuke Zhu, Juan Carlos Niebles, and Li Fei- Fei. RubiksNet: Learnable 3d-shift for efficient video action recognition. In ECCV, pp. 505-521. Springer, 2020.
Christoph Feichtenhofer. X3D: Expanding architectures for efficient video recognition. In CVPR, pp. 203-213, 2020.
Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. SlowFast networks for video recognition. ICCV, pp. 6202-6211, 1, 2019.
Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne West- phat, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitag, et al. The "Something Something" video datahase for learning and evaluating visual common sense. In ICCV, pp. 5842-5850, 2017.
Kensho Hara, Hirokatsu Kataoka, and Yutaka Satoh. Can spatiotemporal 3D CNNs retrace the history of 2D CNNs and ImageNet? In CVPR, pp. 6546-6555, 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Jie Hu, Li Shen, and Gang Sun. Bineerze-and-Excittali networks. In CVPR, pp. 7132-7141, 2018.
Ziyuan Huang, Shiwei Zhang, Liang Pan, Zhiwu Qing, Mingqian Tang, Ziwei Liu, and Marcelo H Ang Jr. Tada! cemporally-adaptins convolutions for video understanding. International Confer- ence on Learning Representations, 2022.
Sergey loffe and Christian Szegedy. Batch normalization:
Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pp. 448-456 PMLR, 2015.
10

---

Published as a conference paper at ICLR 2023
Edwin T Jaynes. Information theory and statistical mechanics. Physical review, 106(4):620, 1957.
Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijaya- narasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action video dataset. arXiv preprint arXiv:1705.06690 06950, 2017.
Dan Kondratyuk, Liangzhe Yuan, Yandong Li, Li Zhang, Mingxing Tan, Matthew Brown, and Bo- qing Gong. MoViNets: Mobile video networks for efficient video recognition. In CVPR, pp. 16020-16030, 2021.
Solomon Kullback. Information theory and statistics. Courier Corporation, 1997.
Kunchang Li, Yali Wang. Gao Peng, Guanglu Song, Yu Liu, Hongsheng Li, and Yu Qiao. Uniformer: Unified transformer for efficient spatial-temporal representation learning. In International Con- ference on Learning Representations.
Yan Li, Bin Ji, Xintian Shi, Jianguo Zhang, Bin Kang, and Limin Wang. TEA: Temporal excitation and aggregation for action recognition. In CVPR, pp. 909-918, 2020.
Ji Lin, Chuang Gan, and Song Han. TSM: Temporal shift module for efficient video understanding. In ICCV, pp. 7083-7093, 2019.
Ming Lin, Hesen Chen, Xiuyu Sun, Qi Qian, Hao Li, and Rong Jin. Neural architecture design for gpu-efficient networks. CoRR, abs/2006. 14090, 2020. URL httpss://axxi....rre.es/a wttp:/iiw.rgggb////o////o
2006 14090
Ming Lin, Pichao Wang, Zhenhong Sun, Hesen Chen, Xiuyu Sun, Qi Qian, Hao Li, and Rong Jin. Zen-NAS: A zero-shot NAS for high-performance image recognition. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 347-356, 2021.
Ze Liu, Jia Ning. Yue Cao, Yixuan Wei, Zheng Zhang, Stephen Lin, and Han Hu. Video swin trans- former. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- fion, pp. 3202-3211, 2022.
Zhaoyang Liu, Limin Wang, Wayne Wu, Chen Qian, and Tong Lu. TAM: Temporal adaptive module for video recognition. In ICCV, pp. 13708-13718, 2021.
Michel Loeve. Probability theory. Courier Dover Publications, 2017.
Ilya Loshchilov and Frank Hutter. SGDR: Stochastic gradient descent with warm restarts. arXiv preprint arXiv: 1608.0383,, 2016.
Joe Mellor, Jack Turner, Amos Storkey, and Elliot Crowley. Neural architecture search without training. In International Conference on Machine Learning, pp. 7588-7598. PMLR, 2021.
Alexander McFarlane Mood. Introduction to the Theory of Statistics. McGraw-Hill, 1950.
Daniel Neimark, Omri Bar, Maya Zohar, and Dotan Asselmann. Video transformer network. In ICCV, pp. 3163-3172, 2021.
Kenneth H Norwich. Information, sensation, and perception. Academic Press San Diego, pp. 81-82, 1993.
Fazlollah M Reza. An introduction information theory. Courier Corporation, 1994.
Andrew M Saxe, Yamini Bansal, Joel Dapello, Madhu Advani, Artemy Kolchinsky, Brendan D Tracey, and David D Cox. On the information bottleneck theory of deep learning. Journal of Statistical Mechanics: Theory and Experiment, 2019(12):124000, 2019.
Claude Elwood Shannon. mathematical theory of communication. The Bell system technical journal, 27(3):379-42) 1948.
Zhenhong Sun, Ce Ge, Junyan Wang, Ming Lin, Hesen Chen, Hao Li, and Xiuyu Sun. Entropy- driven mixed-precision quantization for deep network design. Advances in Neural Information Processing Systems, 2022a.
11

---

Published as a conference paper at ICLR 2023
Zhenhong Sun, Ming Lin, Xiuyu Sun, Zhiyu Tan, Hao Li, and Rong Jin. MAE-DET: Revisiting maximum entropy principle in zero-shot nas for efficient object detection. In International Con- ference Machine Learning, pp. 20810-20826, PMLR, 2022b.
Hidenori Tanaka, Daniel Kunin, Daniel LK Yamins, and Surya Ganguli. Pruning neural networks without any data by iteratively conserving synaptic flow. In Proceedings of the International Conference on Neural Information Processing Systems, 2020.
Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri. Learning spa- tiotemporal features with 3D convolutional networks. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pp. 4489-4497, 2015.
Du Tran, Heng Wang, Lorenzo Torresani, Jamie Ray, Yann LeCun, and Manohar Paluri. closer look at spatiotemporal convolutions for action recognition. In Proceedings ofe the IEEE conference on Computer Vision and Pattern Recognition, pp. 6450-6459, 2018.
Limin Wang, Yuanjun Xiong, Zhe Wang, Yu Qiao, Dahua Lin, Xiaoou Tang, and Luc Van Gool. Temporal segment networks: Towards good practices for deep action recognition. European conference on computer vision, pp. 30-36. Springer, 2016.
Limin Wang, Zhan Tong, Bin Ji, and Gangshan Wu. TDN: Temporal difference networks for effi- cient action recognition. In CVPR, pp. 1895-1904, 202
Zhengwei Wang, Qi She, and Aljosa Smolic. ACTION-Net: Multipath excitation for action recog-
nition. In CVPR, pp. 13214-13223, 2021b.
Zihao Wang, Chen Lin, Lu Sheng. Junjie Yan, and Jing Shao. PV-NAS: Practical neural architecture search for video recognition. arXiv preprint arXiv:2011.00820 2020.
Saining Xie, Chen Sun, Jonathan Huang, Zhuowen Tu, and Kevin Murphy, Rethinking spatiotem- poral feature learning: Speed-accuracy trade-offs in video classification. Proceedings of the European conference computer vision (ECCV), pp. 305-321, 2018.
Yaodong Yu, Kwan Ho Ryan Chan, Chong You, Chaobing Song, and Yi Ma. Learning diverse and discriminative representations via the principle of maximal coding rate reduction. Advances in Neural Information Processing Systems, 33:9422-9434, 2020.
Yuqi Zhang, Chong Liu, Weihua Chen, Xianzhe Xu, Fan Wang, Hao Li, Shiyu Hu, and Xin Zhao. Revisiting instance search: A new benchmark using cycle self-training. Neurocomputing. 501: 270-284, 2022.
Qingin Zhou, Kekai Sheng, Xiawu Zheng, Ke Li, Xing Sun, Yonghong Tian, Jie Chen, and Ron- grong Ji. Training-fres transformer architecture search. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pp. 10894-10908 2022.
12

---

Published as a conference paper at ICLR 2023
APPENDIX
In the appendix, we provide a detailed description of notations in this paper (Appendix A), detailed proof of equations (Appendix B), comparison between different training- scores on the Ima- geNet dataset (Appendix C), a discussion of simple network space in the entropy mechanism (Ap- pendix D), STEntr-Scon for maximizing expressiveness (Appendix E), E3D family structure details (Appendix F), experimental setting details (Appendix G), additional result analysis (Appendix H), and future work discussion (Appendix I).
A MEANING OF NOTATIONS
Notation
Size
Meaning
H(x)
The function of computing the entropy of the given The distribution of the given x
P())
Constant
The value of expectation
The function of computing expectation The value of variance
Constant
I
The function of computing variance Temporal dimension size 3D CNN kernel Height dimension size of 3D CNN kernel
K,
Constant
KR
Constant
K
Constant
Width dimension size of 3D CNN kernel Channel dimension size A 3D CNN kernel size The weight matrix the CNN layer The input feature map size of a given depth (time x height x width) The cosine similarity distance function
Constant K, Kg x K Cx Tx H W
K
S
D
coriie
D
The expanded diversity of cosine similarity distance function
Table 5: The meaning of all notations appeared in this paper.
B PROOF OF SPATIO-TEMPORAL ENTROPY SCORE
B.1 DERIVATION PROCESS OF DIFFERENTIAL ENTROPY
Suppose is sampled from Gaussian distribution N(u, o²), and we know about the probability density function z:
(x - p)
p(x) = V2Ta exp
(9)
20²
can then derive the differential entropy with 1 - dx / as:
H(xx)
llog(p(z)(()ddz
400
(x- - 2
(x - 2)2
expl
llog
expl
20 dr
V2TO
20
/2To
-00
log (mT)
62 v²
dy +
v ? y² y
(10)
V
VT
-log -logV
x -
(0-
c dy)]
1
10g((2)) + log(()) + x log(
2
2
13

---

Published as a conference paper at ICLR 2023
B.2 EXPECTATION AND VARIANCE OF FEATURE MAP
According to Theorem 2 and Theorem 3, we can compute the expectation of 1th layer feature map element ey as:
x!
E(w)
E
J-1 othon
W
Wethoe
B(a othor ) (Wothe)))
(11)
Given two independent random variables i1 and 12 based on (Dv) B(v²) - E(v)² and Theorem we can then calculate the variance of the product of these variables as:
(2) B(o E(u ((o)) ((v,)
- = (v: ((0))) (u2) + ((v)) + (2((0) (B(2))
= (D(v)) + E(vn
(12)
We can then derive the variance of a!. based on Theorem 2 and Theorem 3, as:
K
K
K
(²(a))) D
x athw
W etlo
Kt Kg
i-1 othww w Wathw)
(13)
KC
K
1 (a D (W2
cthw
+ (2 B(W'
( I²
+ DW (Wittw)
B(alahu)) }"}
B.3 PROOF OF 3D CNNS ENTROPY
As the input x is initialized from standard Gaussian distribution (0, 1), and all parameters initialized from Gaussian Distribution (00, oZ),, we can formulate Eq. 11b and Eq. (4) as:
K! KJ
K!
C
B(x!) 0, (x
Σ
D
(W (Woh.n))
(14)
1=1
Subsequently, the expectation B(xt) and variance D22 (x*x;, last layer can derived as:
KF Kf K&
ch-1
B(a+)) = 0, (xtt)
Σ
D (WW...)) }
(15)
Therefore, the variance can be computed by propagating the variances from previous layers as:
(x
II
(16)
1=1
According to Eq. (T). the upper bound entropy is proportional to the variance of last feature map. Then we can derive Eq. as:
H(F)& x Σ
(17)
14

---

Published as a conference paper at ICLR 2023
C DISCUSSION OF SIMPLE NETWORK SPACE
The bias of a convolutional layer is zero, and the activation function in the network is omitted in the search for simplification, following the work of ZenNAS (Lin et al. 202T) and MAE-DET (Sun jet al., 2022b), which has been shown to have no influence on the expressiveness of the network. The training of CNN models has been well studied, and some components can be integrated to boost performance. We deliberately avoid using these components to keep our design simple and universal. Nevertheless, these auxiliary components can easily be plugged into the architecture without any special modifications. Moreover, we provide a discussion of auxiliary components with the entropy calculation, which is listed below.
Batch Normalization (BN). BN is widely used method to re-center and re-scale the features make the network converge faster and more stable. BN normalizes entropies adaptively to the network width (which can be related output variance). When BN is used, networks of different widths will have same entropy value. Hence, BN has to be removed when calculating entropy.
Activation Function. Activation functions increase the non-linearity of training, which has different effects on entropy. For example, ReLUs, half the variance of the output, decrease the entropy with constant factor in each layer, having less positive effect entropy. Mcanwhile, if we formulate each kind activation for our system, it introduces redundancy and becomes complicated, so we give them uniform form omit them in search of concise expressiveness calculation.
Residual Link. the input and all parameters are initialized from standard Gaussian distribution, the variances with without residual links are less than 2% different in entropy score, which means affects the entropy value slightly. Meanwhile, the residual link has a significant impact on conver- gence in training.
Squeeze-and
Sguuezeeand-Exxiitation Excitation Module (SE). SE modules are used to adaptively recalibrate channel-wist feature responses by explicitly modeling interdependency between channels. When the input is initialized from a Gaussian distribution, the output after global pooling in SE block is equal to 0 and the final output becomes 0.5, which will lose the ability model interdependency between channels.
D COMPARISON ON TRAINING-FREB SCORES
D.1 COMPARISON ON THE IMAGENET-H IK DATASET.
Search Devices
Design Cost (hours)
Training-free method
FLOPs
Top-1
ResNet-50
4.1G
78.0
Zen-score (Lin et al. 2021
4.4G
GPU
24
78.9
MAE-DET score (Sun et al. 2022b)
4.4G
GPU
79.1
HomoEntr-Score (w/o Kt)
4.3G
CPU
79.0
Table 6: Comparison of different training-fre free methods ImageNet- 1K dataset. t: Nvidia Tesla V100 16G
GPU, t: AMD Ryzen 5 5600X 6-Core CPU.
We conduct comparison experiments using 2D CNNs on ImageNet with the same evolutionary strategies (ResNet design space), as shown in Table 6 Compared with the result of ResNet-50, the model searched by HomoEntr-Score improves 1.0% accuracy, which indicates that the entropy- based analytic formulation can also measure the information capacity of 2D CNNs. Compared with Zen-score and MAE-DET, the performance of our proposed formulaic metric can also achieve com- parable performance. It means that HomoEntr-See -ctre can work well for modeling the information capacity 2D CNNs, since there no (obvious) discrepancy in the information of the two direc- tions in 2D images statistically.
15

---

Published as a conference paper at ICLR 2023
D.2 COMPARISON ON THE STH-STH VI DATASET.
Since there is no existing code available for training free NAS methods for 3D CNNs, we then refine their implementations for the video recognition task. The results are shown in Table
Search Devices
Design Cost (hours)
Training-free method
GFLOPs
Top-1
Top-5
X3D-S (Feichtenhofer 2020
2G
44.6
74.4
Zen-score (Lin et al.
2021
1.9G
GPU
26
45.5
74.6
MAE-DET score (Sun et al.
2022b
1.9G
GPU
15
45.8
74.7
E3D-S
1.9G
CPU
3
47.1
75.6
Table 7: Comparison of diffcrent training-free methods on the Sth-Sth V1 dataset. t Nvidia Tesla V100 16G
GPU, 1: AMD Ryzen 5 5600X 6-Core CPU.
According to the results in Table 7 the performances of other training-free NAS methods are better than X3D-S, but the performance of our searched model higher. indicates directly applying training-free NAS methods can be effective in the video recognition task, but it still needs spatio- temporal refinement on video understanding tasks, which our work mainly focuses on.
E DETAILED SEARCHING ALGORITHM AND SETTINGS
To obtain highly expressive 3D CNNs of maximized entropy. we use a customized Evolutionary Algorithm. The step-by-step description of EA is given Algorithm as the architecture generator. We only apply the STEntr-Score guide the evolution process, not accuracy, which therefore does not need training on the dataset. We choose EA due to simplicity, and is possible to choose other methods, such as reinforcement learning selection observations, we define the 3D kernel size search space within each layer, (A=*pace)²? ke fitimes (A-*pace)2 to be chosen enable a layer to focus on and aggregate different dimensional representations efficiently, expanding the network's receptive field in the most pertinent directions, while reducing FLOPs along other dimensions (Kondratyuk et al. 2021
even greedy selection. According to our kernel
of the following: {1x3 x3, 1x5x5, 3 3333. These choices
E.1 INITIAL ARCHITECTURE
Stage
Kernels
Channels
Layers
T И W
data
stride 6, 12
13 x 160 x 160
conv1
x 32, 24
24
1
13 x 80 x 80
stage2
[1x12,3x3]
1x1
[48, 48, 24]
1
13 40
40
stageg
[1x12,3x32 1x
[96, 96, 48]
1
13 x 20
20
stage4
[1x12,3x32, 1x12]
[192, 192, 96]
I
13 x 10
10
stages
[1x12,3x3?, 1x12]
[192, 192, 96]
1
13 x 10 x 10
stages
[1x1?, 3x3?,, 1x12]
[384, 384, 192]
1
13 x x 5
conV7
x 12 13 x 5 x
512
1
13 x 5 5
poolg
512
1
1 1
conve/10
1 1x,1111
[2048, #classes]
1x1x1
Table 8: E3D-S initial searching. "stage.." " is super structure which contains "layers"-lane 3D inverted bottleneck block. Channels means the output channels of the corresponding convolution.
Firstly, we set up the initial architecture a MobileNet-styledd network, as shown in Table g which consists of five stages with only layer that can be easily evolved during the algorithm. The
16

---

Published as a conference paper at ICLR 2023
initial architecture is inspired by the structure of X3D-S (Feichtenhofer 2020 because inheriting good prior design can reduce the uncertainty of search space. Then, based on the initial architecture, applied EA helps mutate channel dimension, kernel selection, bottleneck expansion ratio, and layer arrangement by randomly selecting the stage. Note that the channel dimension in conv, and conv, also participate in the mutation process.
Algorithm Maximum Entropy Evolutionary Algorithm
Require: Search space S. Inference budget B, maximal depth L, total number of iterations M, evolutionary population size N, initial structure Fo.
Ensure: Designed E3D backbone F*
1: Initialize population P = {Fo}.
2: for m = 1,2,.... .M do
3: Randomly select Fm E P and select two stages stages E Fm-
4: for = 1,2 do
5: Switch Randomly select one target of (Kernel size, Output channels, Bottleneck chan- nels, Layers} from stagezy do Case kernel: Mutate kernel from 3D kernel search space.
6:
7:
Case Output: Mutate output channels with multiplier space.
8:
Case Bottleneck: Mutate bottleneck channels with expansion ratio space.
9:
Case Layers: Mutate block layers with addend from -,,1,),
end for
11: Get mutated network Fm with two mutated stages stagekj.
12:
if F... is within inference budget B and has no more than L layers then
Get STEntr-Scon of Fm and append F., to
14:
end if'
Remove networks of the smallest STEnt--ccore if the size of P exceeds B.
16:
end for
Return F*, the network of the highest STEntr- in P.
E.2 EVOLUTIONARY ALGORITHM
In Algorithm [] we randomly initialize a population of candidates from the initial structure, under computational budget. The population size and total iterations of EA are set to 512 and 500000, respectively. each iteration step m, we randomly select two stages from the candidates and mutate them. Next, we will randomly select mutation strategy from 4 strategies for each stage. Specific mutation strategies for our E3D family are described as follows. We randomly select 3D kernels from x1x3x3, 1x5x5, 3x3x3} to replace the current one; interchange the expansion ratio bottleneck from (1.5, 2.0, 2.5, 3.0, 3.5, (bottllnle = ratio x intput); scale the output channels with the ratios (2.0, 1.5, 1.25, 0.8, 0.6, 0.5}: increases or decreases depth with 2. Note that the channel dimension of every layer is fixed within from 8 to 640 with multiples of 8, which will help shrink homologous search space and accelerate the search speed. The mutated structure F is appended the population if its inference cost does not exceed the budget. Finally, we maintain the population size by removing networks with the smallest STEntr-Scon -core. After M iterations, the target network with the largest STEntr- is obtained, namely E3D.
F E3D FAMILY ARCHITECTURE DETAILS
Table Table shows three instantiations of E3D with varying complexity, including E3D-S (1.9G FLOPs), E3D-M (4.7G FLOPs), and E3D-L (18.3G FLOPs). All models are searched separately with differ- ent FLOPs budget (1.9G, 4.7G, and 18.4G) for fair comparison with X3D-S/M/L as the baseline. Meanwhile, SE block and ReLU activation function will added into these architectures for train- ing. For both training and inference, the input size remains the same: 160 for E3D-S, 224 for E3D- M, and 312 for E3D-L. All channel dimensions and layer arrangements are searched by evolutionary algorithm under different given budgets.
17

---

Published as a conference paper at ICLR 2023
E3D-S
E3D-M
E3D-L
Stage
filters
output size
filters
output size
filters
output size
data
stride 6,
13 100 x
stride 5, 1-
x 224 221
stride 5,
312 x 312
CODV
x 3'. 24
x 80
3",
x 112 112
3",
156 x 156
12,33
x 12, x 32 1 21
x 12.32
stage2
6
x3
56
x3
16
x 12
1 12.24
x 12, 96
x 12.96
x 12, 120
x 3", 96
X6
13 x 20 20
3 3", ,6
x6
16
28 28
3 3", 120
x13
16
39 39
x 1x.17.18
1 17, 17.61
1 12,18
1 x 12, 176
1x,17. 176
1 176
stagen
3 x
x 10 10
x X6
X
14 14
x 32, 176
x13
16 x
12,
12,
1 12, 120
12,
12,
1x
stages
3 x 32, 176
X
3 x 176
x6
14
3 176
x13
16 20
1 x 120
1 1x112,20
x 1x.12,11
1 x 384
1 161
1 1x11" 480
stages
3 x 384
X6
x
x 3P. 161
X6
1
3 37 180
X13
16 x 10 x 10
x 150
1 184
192
convy
x [2048, #closses]
x
x 464
16x7x7
x 480
16 10 10
pools
x
1x1x1
10
1x1x1 1x1x1
cOnVo/10
1x1x1
[2048, #classes]
1x1x1
[2048. #classes)
Table 9: Three instantiations of E3D with varying complexity. E3D-S with 1.9G FLOPs, E3D-M with 4.7G FLOPs, and E3D-L with 18.4G FLOPs. The size of output is T x H x W.
G EXPERIMENT SETTING DETAILS
G.1 DATASETS
Our experiments are conducted on three large-scale datasets: Something-Somettin (Sth-Sth) &I&V2 (Goyal et al. 2017). and Kinetics400 (Kay et al. 2017). More dataset details can be seen in the supplementary materials. 1) The Sth-Sth datasets
focused on fine mee-grained and motion- dominated actions, which contain pre-defined basic actions involving different interacting objects. Sth-Sth V1 comprises 86k video clips in the training and 12k video clips in the validation set. Sth-Sth V2 is an updated version of Sth-Sth V1, which contains 169k video clips in the training set and 25k video clips in the validation set. They both have 174 action categories. 2) The Kinetics dataset contains activities in daily life and some categories are highly correlated with interacting objects or scene context. Kinetics400 contains over 200k training videos and 20k validation videos divided into 400 categories, covering wide range of human activities.
G.2 IMPLEMENTATION DETAILS
Detailed implementation settings of training & inference stage on Sth-Sth &I& and Kinetics400 datasets are listed in Table 10 All experiments are performed on 8xNvidia Tesla A100 GPUs.
Hyperparameter
Sth-Sth V1&V2
Kinetics400
Epoch
128
256
Batch Size per GPU
32
Optimizer
SGD
SGD
Learning Rate
0.8
0.4
Learning Rate Policy
cosine
cosine
Momentum
0.9
0.9
Weight Decay
5e-5
5e-5
Warm-up Epoch
10
15
Synchronized Batch Normalization
True
True
Training from scratch
True
True
Table 10: List of hyperparameters used Sth-Sth V1&V2 and Kinetics400 datasets.
18

---

Published as a conference paper at ICLR 2023
H ADDITIONAL RESULTS
H.1 ACCURACY VS. COMPLEXITY
Figure shows the trade-off between accuracy and complexity (FLOPs). Compared to 2D CNN- based methods, E3D requires much lower compu- tational resources. Although the performance of our method similar to Tada-R50, the FLOPs Rada-R50 are 4.7 times more than E3D-L. Com- pared to 3D CNN-based methods, we observe that both E3D and MoViNet provement, which indicates that searched methods have higher efficiency in utilizing computing sources. Also, our method achieves comparable performance compared with Mo which indi- cates that the proposed training- free STEntr-Score can effectively evaluate the expressiveness 3D architecture.
CNNS CNNS
achieve large im-
60
80
Figure 5: Accuracy/complexity trade-off on the
Sth-Sth V2 dataset.
H.2 HOMOENTR-SCORE Vs. STENTR-SCORE
Table 11 reports E3D results searched by HomoEntr-See Score and STEntr-Score, under the same search settings. The results show substantial improvement when using STEntr-Score instead of HomoEntr- Score, which indicates the effectiveness of STEntr-Score to handle the discrepancy of visual infor- mation in spatial and temporal dimensions. Even though without refinement factor, the performance of HomoEntr-Sooen Score searched E3D still outperforms X3D, which means the entropy- based search strategy can also measure the expressiveness of 3D CNN architectures.
Model
Resolution
GFLOPs
Top-I
Top-5
X3D-S* [Feichtenhofer, 2020)
13x 160
2
44.6
74.4
MoV (Kondratyuk et al. 2021)
50x 172
2.7
46.9
75.0
E3D (Homoentt-Scorre)
13x x 160
1.9
45.8
74.8
E3D (STEntr-Score)
13x 1602
1.9
47.1
75.6
Table 11: Comparison of different entropy scores the Sth-Sth V1 dataset.
denotes our reproduced models.
H.3 3D KERNEL SEARCH SPACE
To analyze the impact of kernel search space, we expand the 3D kernel search space and conduct experiments, as shown in Table [2 The results indicate that larger search spaces actually benefit the performance, However, compared to the results between E3D (HomoEntr-Scorry with E3D (STEntr Score)) in Table the STEntr-Scort Score based searching can boost the performance (+1.3%) 1,3%) more than a large search space did (+0.2%). It also verified the effectiveness of our proposed STEntr-Sconn in evaluating the expressiveness
CNNs.
Kernel Search Space
FLOPs
Top-1
STEntr-Scora
1 1x3x3, 3x3x3 1x3x3, 1x5x5, 3x3x3 1x3x3, 1xxx,, 3x3x3, 3x
1.9G
46.3
198.55
1.9G
47.1
202.86
1.9G
47.1
202.74
1x3x3, 1x5x5, 3x3x3, 5x3x3
1.9G
47.2
203.13
Table 12: Comparison of different 3D kernal scarch space on the Sth-Sth V1 datasct.
19

---

Published as a conference paper at ICLR 2023
H.4 INFERENCE TIME COMPARISON
We report the inference time comparison with some state-of-the-art methods in Table All models trained and tested the Sth-Sth VI dataset, and the batch size is 16. Compared to X3D, our E3D performs better not only on accuracy but also costs lower inference time. It indicates that the searched architecture by our proposed STEntr-Score is more effective and efficient for video understanding, Compared to MoViNet, even though Top-1 accuracies are similar, both latency and throughput of E3D are performing better. Due to Mo Viee applies a causal convolutional network and contains more parameters. Compared to 2D CNN-based methods, E3D performs better on both accuracy and running time and requires much lower computational resources. Overall, we believe that our proposed E3D family is more efficient and practical for real-world applications.
Method
Resolution
Frame
GFLOPs
#Param Topl
Latency (ms/video) Throughput(vdeoos)
TSM et 2019
256
16
23.9M
47.2
23.0
43.5
TANet et
256
16
66
26M
47.6
14.7
68.0
X3D-M Feichtenhhofr 2020
224
16
4.7
3.7M
47.3
13.5
74.1
MoViNet-A (Kondratykk et al. 2021
172
50
6
4.6M
49.3
21.9
45.7
E3D-M
224
16
4.7
3.4M
49.4
11.4
87.7
Table 13: Inference comparison using a Tesla V100 on the Sth-Sth dataset.
I FUTURE DIRECTION
Data-driven design. The design of STEntr- search correlates with parameter initialization and kernel selection, with standard Gaussian initialization input. If we replace the Gaussian input directly with target data, the output after a convolution will be random due to the Gaussian initialized weights, the process of ST'ntt-SSoor based searching is contained without data training. The aim of our work is therefore to provide a training- fiining-ffree approach to 3D CNN architecture design according the maximum entropy principle under the given budgets. We believe that the training- method, combined with target data without training, could be future direction for research.
Transformer model. We believe that the principle of maximum entropy is theoretically applicable to transformers. However, there exist some challenges to overcome. For example, Transformer has more complex components than CNN, such as 'Q' and 'K' kernel operation and melti-head attention, which is difficult to calculate the maximum entropy. In addition, the discrepancy of visual information in spatial and temporal dimensions by Transformer still remains challenge. Although these challenges are difficult to overcome, this would be fascinating task for in the future.
20


Reviews:
[{"review_id": "rC91lzOYho", "reviewer": "Reviewer_SqC4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper aims to find a good 3D CNN architecture for video recognition. This study proposes to use a novel proxy task, maximizing a designed entropy value, to search for the effective architecture. The experiments show the effectiveness of the proposed algorithm.\n", "summary_of_the_review": "This paper is clear. The story makes sense. The results of this paper is good, and the author shows that the proposed method is better than random search. However, it makes the paper less convincing if don't compare with other NAS based methods.", "strength_and_weaknesses": "Strength\n1. The experiments are solid, showing a good alignment between the proxy task and the video recognition ability.\n2. The results on Sth-Sth are good.\n\nQuestion\n1. “Table 9 shows three instantiations of E3D with varying complexity, including E3D-S (1.9G FLOPs), E3D-M (4.7G FLOPs), and E3D-L (18.3G FLOPs).” Wondering the details about how to get E3D-S/M/L – are they searched separately? Or search a model and then manually scale up.\n2. Is it possible to get an even larger model (maybe the name would be E3D-XL)? If it is possible, would we expect higher performance?\n3. Is it possible to extend this entropy idea to other architectures (eg transformer)?\n4. Is it possible to trivially apply other training-free NAS on 3D task? I am a little bit concern that it is not convinced the proposed method is better than previous training-free NAS method.\n\nWeakness\n1. The results on Kinetics400 are not as good as the transformer based method (although I understand the transformers are computationally expensive)\n2. See question 4: It seems mainly compared with manually designed architectures, but don't compare with other NAS methods.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "n8D6KYZ1b1d", "reviewer": "Reviewer_Jkpo", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a spatio-temporal entropy score to conduct training-free neural architecture search for efficient video recognition. They use a simple network search space with a carefully designed performance estimation score. The searched results achieve state-of-the-art performance on action recognition.", "summary_of_the_review": "Baselines are most 2D CNNs-based methods and outdated. So I recommend to weak reject this paper until results on stronger baselines are provided.", "strength_and_weaknesses": "## Strength\n\n1. The derivation of the entropy score is thorough.\n1. The visualization of the searched architectures in Figure 1 reveals interesting properties of 3D CNNs.\n\n## Weakness\n\n1. As the search space is the most important part of a NAS method, the authors should give more details such as the lower bound / upper bound performance of the search space.\n1. The search happens without considering auxiliary modules like BN, Reslink, and so on. Will the entropy score transfer well with such a gap?\n1. Why is Table 2 only compared with most 2D CNN-based methods with ResNet50? Also, the baselines are outdated.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "4yumpVQ0Y_", "reviewer": "Reviewer_RjKM", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed a spatio-temporal entropy score (STEntr-Score) with a refinement factor to handle the discrepancy of visual information in spatial and temporal dimensions, through dynamically leveraging the correlation between the feature map size and kernel size depth-wisely. Then the proposed entropy-based 3D CNNs (E3D family), can then be efficiently searched by maximizing the STEntr-Score under a given computational budget, via the existing evolutionary algorithm without training the network parameters. Comprehensive comparisons on the popular benchmark datasets show the advances of the proposed method. ", "summary_of_the_review": "Overall, I think this is a good paper which will have positive contributions to this field. ", "strength_and_weaknesses": "The strengths of the paper are: \n+ This paper tried to exploit the important direction for the video action recognition, or generally the video intellengent analysis. \n+ The authors proposed to estimate correlation between feature map and kernel size in different depths, and furthermore, they proposed the Spatio-Temporal Entropy Score. \n+ The proposed STEntrScore is positively correlated with Top1 accuracy which indicates that the proposed spatio-temporal refinement can handle the discrepancy of visual information in spatial and temporal dimensions.\n+ The performance on the benchmark datasets is promising for the research in this area.\n+ Inference and training speed are appealing \n\nThe weaknesses of the paper are: \n- No limitation discussion in the paper \n- I would like to see some discussions on the affect of the temporal resolution on the performance, such as if we increase or decrease the number of frames we consider in the searching and final architecture, what the performance? \n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster