AssembleNet: Searching for Multi-Stream Neural Connectivity in Video Architectures
Authors: Michael S. Ryoo
AJ Piergiovanni
Mingxing Tan
Anelia Angelova
Venue:  (2020)
Keywords: video representation learning
video understanding
activity recognition
neural architecture search

Abstract: Learning to represent videos is a very challenging task both algorithmically and computationally. Standard video CNN architectures have been designed by directly extending architectures devised for image understanding to include the time dimension, using modules such as 3D convolutions, or by using two-stream design to capture both appearance and motion in videos. We interpret a video CNN as a collection of multi-stream convolutional blocks connected to each other, and propose the approach of automatically finding neural architectures with better connectivity and spatio-temporal interactions for video understanding. This is done by evolving a population of overly-connected architectures guided by connection weight learning. 
Architectures combining representations that abstract different input types (i.e., RGB and optical flow) at multiple temporal resolutions are searched for, allowing different types or sources of information to interact with each other. Our method, referred to as AssembleNet, outperforms prior approaches on public video datasets, in some cases by a great margin. We obtain 58.6% mAP on Charades and 34.27% accuracy on Moments-in-Time.

Published as a conference paper at ICLR 2020
ASSEMBLENET: SEARCHING FOR MULTI-STREAM NEURAL CONNECTIVITY IN VIDEO ARCHITECTURES
Michael S. Ryoo! AJ Piergiovanni! Mingxing Tan & Anelia Angelova! 1.2
Robotics at Google
?Gogge Research
SIndiana University Bloomington
mryoo@google.coom com
ABSTRACT
Learning to represent videos is a very challenging task both algorithmically and computationally. Standard video CNN architectures have been designed by directly extending architectures devised for image understanding to include the time di- mension, using modules such as 3D convolutions, or by using two-stream design to capture both appearance and motion in videos. We interpret video CNN as a collection of multi-stream convolutional blocks connected to each other, and propose the approach of automatically finding neural architectures with better connectivity and spatio-temporal interactions for video understanding. This is done by evolving population of overly- weerly-connected architectures guided by con- nection weight learning. Architectures combining representations that abstract different input types (i.e., RGB and optical flow) at multiple temporal resolutions are searched for, allowing different types or sources of information to interact with each other. Our method, referred to as AssembleNet, outperforms prior approaches on public video datasets, in some cases by a great margin. We obtain 58.6% mAP on Charades and 34.27% accuracy on Moments-in-Time
INTRODUCTION
Learning to represent videos is challenging problem. Because video contains spatio-temporal data, its representation is required to abstract both appearance and motion information. This is particularly important for tasks such activity recognition, as understanding detailed semantic contents of the video is needed. Previously, researchers approached this challenge by designing a two-stream model for appearance and motion information respectively, combining them by late or intermediate fusion to obtain successful results: [Simonyan & Zisserman (2014); Feichtenhofer et al. (2]((2006)) a 2017 2018). However, combining appearance and motion information is open problem and the study on how and where different modalities should interchange representations and what temporal aspect/resolution each stream (or module) should focus on has been very limited.
In this paper, we investigate how to learn feature representations across spatial and motion visual clues. We propose guided evolution, which focuses on finding higher-level connectivity between network blocks taking multiple input streams different temporal resolutions. Each block itself is composed of multiple residual modules with space-time convolutional layers, learning spatio-temporal representations, Our architecture learning not only considers the connectivity between such multi-stream, multi-resolution blocks, but also merges and splits network blocks to find better multi-stream video CNN architectures. Our objective is to address two main questions in video representation learning: (1) what feature representations are needed at each intermediate stage of the network and at which resolution and (2) how to combine exchange such intermediate representations (i.e., connectivity learning). Unlike previous neural architecture search methods for images that focus on finding good 'module' of convolutional layers be repeated in single-stream networks (Zoph al., 2018 Real et al. 2019). our objective is to search for higher-level connections between multiple sequential or concurrent blocks to form multi-stream architectures.
multi-stream neural architecture search algorithm with connection learning
We propose the concept of AssembleNet, a new method of fusing different sub-networks with different input modalities and temporal resolutions. AssembleNet is general formulation that

---

Published as a conference paper at ICLR 2020
C-512
[-2
256 rml
r=4
Ca
C=
C-
Ca64
Repested in time
Cm330 32
C=32
C=
C=332 1=2
rc
F24
r=l
RGB
Flow
Figure 1: AssembleNer with multiple intermediate streams. Example learned architecture. Darker colors of connections indicate stronger connections. At each convolutional block, multiple 2D and (2+1)D residual modules are repeated alternatingly. Our network has block levels (+ the stem level connected to raw data). Each convolutional block has its output channel size (i.e., the number of filters) C' and the temporal resolution r' controlling the ID temporal convolutional layers in
allows representing various forms of multi-stream CNNs as directed graphs, coupled with an efficient evolutionary algorithm to explore the network connectivity. Specifically, this is done by utilizing the learned connection weights guide evolution, in addition randomly combining, splitting, or connecting sub-network blocks. AssembleNet is a "family' of learnable architectures; they provide generic approach to learn connectivity among feature representations across input modalities, while being optimized for the target task. We believe this is the first work to (i) conduct research on automated architecture search with multi-stream connections for video understanding, and (ii) introduce the new
evolutionary algorithm for neural architecture search.
Figure D shows an example learned AssembleNet. The proposed algorithm for learning video architectures is very effective: it outperforms all prior work and baselines on two very challenging benchmark datasets, and establishes new state-of-the-arr, AssembleNet models equivalent number parameters to standard two-stream (2+1)D ResNet models.
2 PREVIOUS WORK
video is spatio-temporal data (i.e., image frames concatenated along time axis), and its rep- resentation must abstract both spatial and temporal information. Full 3D space-time (i.e., XYT) convolutional layers as well as (2+1)D convolutional layers have been popularly used represent videos (Tran et al. 22014] Carreira & Zisserman 2017; Tran et al. [0018] Xie et al. 2018). Researchers studied replacing 2D convolutional layers in standard image-based CNNs such as Inception (Szegedy ct al. 2016 and ResNet (He et al. 2016), so that it can be directly used for video classification.
Two-stream network designs, which combine motion and appearance inputs, are commonly used (e.g.. Simonyan & Zisserman 2014 Feichtenhofer et al., [2016a] 2017] 20166). Combining appearance information two different temporal resolutions (e.g., :. 24 vs. frames per second) with intermediate connections has been proposed by |eichhennhofrr et al. (2018). Late fusion of the two-stream representations architectures with more intermediate connections (Diba et al. 2019), have also been explored. However, these video CNN architectures are the result of careful manual designs by human experts.
Neural Architecture Search (NAS), the concept automatically finding better architectures based on data, is becoming increasingly popular (Zoph & Le 2017 Zoph al.) 2018 Liu al.) 2018). Rather than relying on human expert knowledge to design a CNN model, neural architecture search allows the machines generate better performing models optimized for the data. The use of reinforcement learning controllers (Zoph & Le| 2017; Zoph et al., (218)) as well as evolutionary algorithms (Real al. 2019) have been studied, and they meaningfully outperform handcrafted architectures. Most of
2

---

Published as a conference paper at ICLR 2020
these works focus on learning architectures of modules (i.e., groupings of layers and their connections) to be repeated within a fixed single-strean meta-architecturr (c.g., ResNet) for image-based object classification. One-shot architecture search to learn differentiable connections (Bender al. 2018) Liu et al.) 2019) has also been successful for images. However, it is very challenging to directly extend such work to find multi-stream models for videos, as it requires preparing all possible layers and interactions the final architecture may consider using. In multi-stream video CNNs, there are many possible convolutional blocks with different resolutions, and fully connecting them requires significant amount of memory and training data, which makes it infeasible.
Our work is also related to Ahmed & Torresani (201)) which used learnable gating to connect multiple residual module branches, and to the kandWire network (Xie et al. 2019), which showed that randomly connecting a sufficient number of convolutional layers creates performant architectures. However, similar to previous NAS work, the latter focuses only on generating connections between the layers within block. The mnchtarchitecture is fixed as single stream model with a single input modality. In this work, our objective is to learn high-level connectivity between multi-stream blocks for video understanding driven by data. We confirm experimentally that in the multi-stream video CNNs, where multiple types of input modalities need to be considered at various resolutions, randomly connecting blocks is insufficient and the proposed architecture learning strategy is necessary.
3 ASSEMBLENET
We propose a new principled way to find better neural architectures for video representation learning. We first expand video CNN to multi-resolution, multi-stream model composed of multiple sequential and concurrent neural blocks, and introduce a novel algorithm to search for the optimal connectivity between the blocks for a given task.
We model a video CNN architecture as a collection of convolutional blocks (i.e., sub-networks) connected to each other. Each block is composed of residual module of space-time convolutional layers repeated multiple times, while having its own temporal resolution. The objective of our video architecture search to automatically (I) decide the number of parallel blocks (i.e., how many streams to have) at each level of the network, (2) choose their temporal resolutions, and (3) find the optimal connectivity between such multi-stream neural blocks across various levels. The highly interconnected convolutional blocks allow learning of the video representations combining multiple input modalities various temporal resolutions. We introduce the concept of connection-leamnin guided architecture evolution to enable multi-stream architecture search.
We name our final architecture 'AssembleNet', since it is formulated by assembling (i.e., merging, splitting, and connecting) multiple building blocks.
3.1 GRAPH FORMULATION
In order to make our neural architecture evolution consider multiple different streams with different modalities at different temporal resolutions, we formulate the multi-stream model as a directed acyclic graph. Each node in the graph corresponds to a sub-network composed of multiple convolutional layers (i.e., a block), and the edges specify the connections between such sub-networks. Each architecture denoted as G, (N, E.) where N;- = (noi, 7i6s ... } is the set of nodes and E, is the set of edges defining their connectivity.
Nodes. A node in our graph representation is a ResNet block composed of a fixed number of interleaved 2D and (2+1)D residual modules. A '2D module' is composed of a 1x1 conv. layer, one 2D conv. layer with filter size 3x3, and one 1xl convolutional layer. A '(2+1)D module' consists of a temporal 1D convolutional layer (with filter size 3), a 2D conv. layer, and 1x1 conv. layer. In each block, we repeat a regular 2D residual module followed by the (2+1)D residual module m times.
Each node has its own block level, which naturally decides the directions of the edges connected it. Similar to the standard ResNet models, we made the nodes have a total of four block levels (+ the stem level). Having multiple nodes of the same level means the architecture has multiple parallel 'streams'. Figure illustrates an example. Each level has a different m value: 1.5, 2, 3, and 1.5. m 1.5 means that there is one 2D module, one (2+1)D module, and one more 2D module. As result, the depth of our network is 50 conv. layers. We also have a batch normalization layer followed by ReL.U after every conv. layer.
3

---

Published as a conference paper at ICLR 2020
There are two special types of nodes with different layer configurations: source nodes and sink nodes. source node in the graph directly takes the input and applies small number of convolutional/pooling layers (it is often referred as the 'stem' of a CNN model). In video CNNs, the input is a 4D tensor (XYT channel) obtained by concatenating either RGB frames or optical flow images along the time axis. Source nodes are treated as level-0 nodes. The source node is composed of one 2D conv. layer of filter size 7x7, one ID temporal conv. layer of filter size 5, and one spatial max pooling layer. The ID conv. is omitted in optical flow stems. A sink node generates the final output of the model, and it is composed of one pooling, one fully connected, and one softmax layer. The sink node is also responsible for combining the outputs of multiple nodes at the highest level, by concatenating them providveed after the pooling. More details are provided in Appendix.
Each node in the graph also has two attributes controlling the convolutional block: its temporal resolution and the number of channels. We use temporally dilated ID convolution to dynamically change the resolution of the temporal convolutional layers in different blocks, which are discussed more below. The channel size (i.c.. the number of filters) of node could take arbitrary values, but we constrain the sum of the channels of all nodes in the same block level to be constant that the capacity of an AssembleNet model is equivalent to a ResNet model with the same number of layers.
Temporally Dilated ID Convolution. One of the objectives is to allow the video architectures to look at multiple possible temporal resolutions. This could be done by preparing actual videos with different temporal resolutions as in FFeichtenhooer et al. (2018) or by using temporally 'dilated convolutions as we introduce here. Having dilated filters allow temporal ID conv. layers to focus on different temporal resolution without losing temporal granularity. This essentially is a ID temporal version of standard 2D dilated convolutions used in Chen et al. 2018) or Yu & Koltun (016))
Let & be a temporal filter (i.c., a vector) with size 2d + 1. The dilated convolution operator *, is similar to regular convolution but has different steps for the summation, described as:
(F *, k)(()=
= Σ F(1))((12+ + dd)
A1 4rtat
where I, (,, and l2 are time indexes. r' indicates the temporal resolution (or the amount of dilation), and the standard 1D temporal convolution is special case where r 1. In the actual implementation, this is done by inserting r - number of zeros between each element of & to generate k', and then convolving such zero-inflated filters with the input: F *, k F * k.. Importantly, the use of the dilated convolution allows different intermediate sub-network blocks (i.e., not just input stems) to focus on very different temporal resolutions at difffrrent levels of the convolutional architecture.
Note that our temporally dilated convolution is different from the one used in Lea et al. (2017), which designed a specific layer to combine representations from different frames with various step sizes. Our layers dilate the temporal filters themselves. Our dilated convolution can be viewed as direct temporal version of the standard dilated convolutions used in Chen et al. (2018) ; Yu & Koltun (2016).
Edges. Each directed edge specifies the connection between two sub-network blocks, and it de- scribes how a representation transferred from one block to another block. We constrain the direction of each edge so that it is connected from lower level block to a higher level block to avoid forming a cycle and allow parallel streams. A node may receive inputs from any number of lower-level nodes (including skip connections) and provide its output to any number of higher-level nodes.
Our architectures use (learnable) weighted summation to aggregate inputs given from multiple connected nodes. That is, an input to a node is computed as F = E, sigmoid(w;) FF where Fout are output tensors (i.e., representations) of the nodes connected to the node and w; are their corresponding weights. Importantly, each w; is considered training data through back propagation. This has two key advantages compared to conventional feature map concatenation: (i) The input tensor size is consistent regardless of the number of connections. (ii) We use learned connection weights to "guide" our architecture evolution algorithm in a preferable way, which we discuss more in Section 3.2
variable that has to be learned from
If the inputs from different nodes differ in their spatial size, we add spatial max pooling and striding to match their spatial size. If the inputs have different channel sizes, we add 1xl conv. layer match the bigger channel size. Temporal sizes of the representations is always consistent in our graphs, as there is no temporal striding in our formulation and the layers in the nodes are fully convolutional.
4

---

Published as a conference paper at ICLR 2020
Round 17 Fitness m 60.60 60.68
Round 37
Round 57
Round - 83 Fimess 62, 16
Round - 99
Fimess 62.25
Fimess 61.91
Fimeess 64.02
Figure 2: An example showing sequence of architecture evolution. These architectures have actual parent-child relationships. The fitness of the third model was worse than the second model (due to random mutations), but it was high enough to survive in the population pool and eventually evolve into a better model.
3.2 EVOLUTION
We design an evolutionary algorithm with discrete mutation operators that modify nodes and edges in architectures over iterations. The algorithm maintains a population of P different architectures, P {G,, G2..... G J. where each architecture G is represented with set of nodes and their edges as described above.
The initial population is formed by preparing a fixed number of randomly connected architectures (e.g.. P|| = 20). Specifically, we (1) prepare a fixed number of stems and nodes at each level (e.g.. two per level), (2) apply number of node split/merge mutation operators which discuss more below, and (3) randomly connect nodes with the probability 0.5 while discarding architectures with graph depth 4. As mentioned above, edges constrained so that there is directed edge reversing the level ordering. Essentially, a set of overly-connected architectures are used as starting point. Temporal resolutions are randomly assigned to the nodes.
We use the tournament selection algorithm (Goldberg & Deb 1991) as the main evolution framework: At each evolution round, the algorithm updates the population by selecting a 'parent' architecture and mutating (i.e., modifying) it to generate a new 'child' architecture. The parent is selected by randomly sampling subset of the entire population P' = P, and then computing the architecture with the highest "fitness": Gp argmax f f'G.)) where f(G) is the fitness function. Our fitness is defined video classification accuracy of the model, measured by training the model with certain number of initial iterations and then evaluating it on the validation set as its proxy task. More specifically,
t accuracy + top-5 accuracy as the fitness function. The child is added into the population, and the model with the least fitness is discarded from the population.
A child is evolved from the parent by following two steps. First, it changes the block connectivity (i.e., edges) based on their learned weights: cannection-learning learning-guided evolution', Next, it applies random number of mutation operators to further modify the node configuration. The mutation operators include (1) a random modification of the temporal resolution of a convolutional block (i.e., node) as well (2) merge or split of block. When splitting node into nodes, make their input/output connections identical while making the number of channels in their convolutional layers half that of the node before the split (i.e., C = C2/2 where Cp is the channel size of the parent). More details are found in Appendix. As a result, we maintain the total number of parameters, since splitting or merging does not change the number of parameters of the convolutional blocks.
Connection- Leaaninn-Guuide Mutation. Instead of randomly adding, removing or modifying block connections to generate the child architecture, we take advantage of the learned connection weights from its parent architecture. Let E, be the set of edges of the parent architecture. Then the edges of the child architecture E, are inherited from Ep. by only maintaining high-weight connections while replacing the low-weight connections with new random ones. Specifically, E, = E!! U U.:
E (ee E, W. B). E fo E (E. E,)
E, - E1
X
E
where ~ (0, 1) and E, is the set of all possible edges. Eg corresponds to the edges the child architecture inherits from the parent architecture, decided based on the learned weight of the edge W..
5

---

Published as a conference paper at ICLR 2020
C= r=2
C1512
C=512
= 125 r=2
== 128 F=
C=
C 128 r-4
C= 128 r-
C 128 r-4
C= 128 r-8
C- 128 1=4
C- 128 r=2
C- 12X r=8
C- 128
r=2
r=4
128
128
C4
C= 64 rw8
C=64
C=
C52 r=2
[=1
F=4
2=1
2=
r=
==
C= 64
C=
C= 32
C= 32
C= 32
C=
C=
=
C=
32
r-
RGB
Flow
RGB
Flow
RGB
Flow
Figure 3: More AssembleNet examples. Similarly good performing diverse architectures, all with higher-than-500 500% mean-average precision on Charades. For instance, even our simpler two-stem AssembleNet-50 (left) got 51.4% mAP on Charades. Darker edges mean higher weights.
This is possible because our fitness measure involves initial proxy training of each model, providing the learned connection weight values Wa of the parent.
B. which controls whether or to keep an edge from the parent architecture, could either be constant threshold or a random variable following uniform distribution: B = b or B = XB ~ u0if(0, 1). E? corresponds the new randomly added edges which were We enumerate through each possible new edge, and randomly add it with the probably of EE / Fpl- This makes the expected total numher added edges to be F. - F maintaining the size of E... Figure[]shows example of the evolution process and Figure shows final architectures.
the parent architecture.
Evolution Implementation Details. Initial architectures are formed by randomly preparing either {2 or 4} stems, two nodes per level at levels I to 3, and one node level 4. We then apply 1~5 random number of node split operators so that each initial architecture has a different number of nodes. Each node is initialized with random temporal resolution of 1, 2, 4, or 8. As mentioned, each possible connection then added with the probability of 0.5.
At each evolution round, the best-performing parent architecture is selected from random subset of 5 from the population. The child architecture is generated by modifying the connections from the parent architecture (Section 3.2 A random number of node split, merge, temporal resolution change mutation operators (0~o4) are then applied. Evaluation of each architecture (i.e., measuring the fitness) is done by training the model for 10K iterations and then measuring its top-1 top-5 accuracy on the validation subset. The Moments-in- Time dataset, described in the next section, used as the proxy dataset to measure fitness. The evolution was run for 200 rounds, although good performing architecture was found within only 40 rounds (c.g., Figure right). Figure [ shows the model found at the 165th round. 10K training iterations of each model during evolution took 3~55 hours; with our setting, evolving a model for rounds took less than a day with parallel workers.
EXPERIMENTS
4.1 DATASETS
Charades Dataset. We first test on the popular Charades dataset (Sigurdsson et al. 2016) which is unique in the activity recognition domain as it contains long sequences. It is one of the largest public datasets with continuous action videos, containing 9848 videos of 157 classes (7985 training and 1863 testing videos). Each video is ~00 seconds. It is challenging dataset due the duration and variety of the activities. Activities may temporally overlap in a Charades video, requiring the model predict multiple class labels per video. We used the standard 'Charades v1 classify' setting for the evaluation. To comply with prior work (e.g. Feichtenhofer et al.) 2018). we also report results when pre-training Kinetics (Carreira & Zisserman 2017), which another large-scale dataset.

---

Published as a conference paper at ICLR 2020
Table Reported state-of-the-art action classification performances (vs. AssembleNet) on Charades.
"2-stream (2+1)D ResNet-50' is the two-stream model with connection learning for level-4 fusion.
Method
pre-train
modality
mAP
2-stream Simonyyn & Zisserman
2014
UCFI01
RGB+Flow
18.6
Asyn-TF Sigurdsson et al. 2017)
UCF101
RGB+Flow
22.4
CoViAR (Wu ct al. 2018b
ImageNet
Compressed
21.9
MultiScale TRN (Zhou et al. 2018
ImageNet
RGB
25.2
13D (Carreira & Zisserman
2017)
Kinetics
RGB
32.9
I3D (trom Wang (2018)
2018
Kineties
RGB
35.5
13D-NL Wang et al.
Kineties
RGB
37.5
STRG (Wang Gupta (0188)
Kineties
RGB
39.7
LFB-101 (Wu et al.
| 2018a)
Kinetics
RGB
42.5
SlowFast- IUI (Feichtenhoter al. 2018
Kineties
RGB+RGB
45.2
2-stream (2+1)D ResNet-50
MiT
RGB+Flow
48.7
2-stream (2+1)D ResNet-50 (ours)
Kinetics
RGB+Flow
50.4
2-stream (2+1)D ResNet-101 (ours)
Kinetics
RGB+Flow
50.6
AssembleNet-50 (ours)
MiT
RGB+Flow 53.0
AssembleNet-50 (ours)
Kinetics
RGB+Flow
56.6
AssembleNet-101 (ours)
Kinetics
RGB+Flow
58.6
Table 2: State-of-the-art action classification accuracies on Moments in Time (Monfort et al. 2018).
0.63
0.61
Method
modality
Top-1
Top-5
ResNet50-ImageNNa
RGB
27.16
51.68
Fitness
TSN (Wang ct al. 2016)
RGB
24.11
49.10 10
0.59
loffe & Szegedy
2013)
Flow
11.60
27.40
0.58
TSN-Flow Wang ct
2016
Flow
15.71
34.65
TSN-2Stream (Wang
2016
RGB+F
25.32
50.10
Connection lsaming guided
TRN-Multi [Zhou el al.
2018
RGB+F
28.27
53.87
0.56
Standard evalution
Two-stream (2+1)D I ResNet-50
RGB+F 28.97
55.55
Random
I3D (Carreira & Zisserman
2017
RGB+F 29.51
56.06
10
20
30
40
50
60
70
Evolution round
AssembleNet-s0
RGB+F 31.41
58.33
AssembleNet-Su (with Kinetics)
RGB+F 33.91
60.86
Figure 4: Comparison of different search methods.
AssembleNet-100 101 (with Kinetics)
RGB+F 34.27
62.71
We note that Kinetics is shrinking in size (~155% videos removed from the original Kinetics-400) and the previous versions are no longer available from the official site.
Moments in Time (MiT) Dataset. The Moments in Time (MiT) dataset (Monfort et al. 2018) is a large-scale video classification dataset with more than 800K videos (~33 seconds per video). It is very challenging dataset with the state-of-the-art models obtaining less than 30% accuracy. We use this dataset for the architecture evolution, and train/test the evolved models. We chose the MiT dataset because provides sufficient amount of training data for video CNN models and allows stable comparison against previous models. We used its standard classification evaluation setting.
4.2 RESULTS
Tables and 2 compare the performance of AssembleNet against the state-of-the-e models. We denote AssembleNet more specifically as AssembleNet-500 since its depth is 50 layers and has an equivalent number of parameters to ResNet-50. AssembleNet-11 1 is its 101 layer version having equivalent number of parameters to ResNet-101 AssembleNet is outperforming prior works on both datasets, setting new state-of-the-ar results for them. Its performance MÎT is the first above 34%. We also note that the performances on Charades is even more impressive at 58.6 whereas previous known best results are 42.5 and 45.2. For these experiments, the architecture search was done the MiT dataset, and then the found models are trained and tested on both datasets, which demonstrates that the found architectures useful across datasets.
7

---

Published as a conference paper at ICLR 2020
Table 3: Comparison between AssembleNet and ar- chitectures without evolution, but with connection weight learning. Four-stream models are reported here for the first time, and are very effective. All these models have a similar number of parameters.
Table 4: Ablation comparing different Assem- bleNet architectures found with full vs. con- strained search spaces. The models are trained from scratch.
Architecture
MiT Charades
Architecture
MiT
Two-stream (late fusion)
28.97
46.5
Baseline (random conn. learning)
29.91
Two-stream (fusion at lv. 4)
30.00
48.7
No mutation
30.26
Two-stream (flow>RGB inter.)
30.21
49.5
RGB-only
30.30
Two-stream (fully, fuse at 4)
29.87
50.5
Without temporal dilation
30.49
Four-stream (fully, fuse 4)
29.98
50.7
Two-stem only
30.75
Random (+ connection learning)
29.91
50.1
Full AssembleNet-50
31.41
AssembleNet-50
31.41
53.0
In addition, compare the proposed connection-leamnine guideed evolution with random architecture search and the standard evolutionary algorithm with random connection mutations. We made the standard evolutionary algorithm randomly modify 1/3 of the total connections each round, that roughly the number of edges the connection- connection-eerring the results, visualizing the average fitness score of the three top- peppefforming models in each pool. We observe that the manection-learning guided evolution is able to find better architectures, and is able to do so more quickly. The standard evolution performs similarly to random search and is not as effective. We believe this is due to the large search space the approach is required to handle, which is exponential to the number of possible connections. For instance, if there are N nodes, the search space complexity is 20(N") just for the connectivity search. Note that the initial ~330 rounds are always used for random initialization of the model population, regardless of the search method.
evolution modifies. Figure 4 shows
4.3 ABLATION STUDIES
We conduct an ablation study comparing the evolved AssembleNet multiple (2+1)D two-stream (or multi-stream) architectures which are designed to match the abilities of Assemblenet but without evolution. We note that these include very strong architectures that have not been explored before, such as the four-stream model with dense intermediate connectivity. We design competitive networks having various connections between streams, where the connection weights also learned (see the supplementary material for detailed descriptions and visualizations). Note that all these models have equivalent capacity (i.e., number of parameters). The performance difference due to network structure. Table 3 shows the results, demonstrating that these architectures with learnable inter- connectivity are very powerful themselves and evolution is further beneficial. The Moments in Time models were trained from scratch, and the Charades models were pre-trained on MiT. In particular, we evaluated architecture with intermediate connectivity from the flow stream RGB, inspired by FFeihhennoofrr et al. (2016b 2018) (+ connection weight learning). It gave 30.2% accuracy on MiT and 49.5% on Charades, which are not accurate as AssembleNet. Randomly generated models (from 50 rounds of search) are also evaluated, confirming that such architectures do not perform well.
Further, we conduct another ablation confirm the effectiveness of search space. Table compares the models found with our full search space vs. more constrained search spaces, such as only using two stems and not using temporal dilation (i.e., fixed temporal resolution).
4.4 GENERAL FINDINGS
As the result of conmection-eerming-guiiee architecture evolution, non-obvious and non-intuitive connections found (Figure 3 As expected, more than one possible "connectivity" solution can yield similarly good results. Simultaneously. models with random connectivity perform poorly compared to the found AssembleNet. Our observations also include: (1) The models prefer have only one block at the highest level, although we allow the search to consider having more than one block that level. (2) The final block prefers simple connections gathering all outputs of the blocks in the 2nd to last level. (3) Many models use multiple blocks with different temporal resolutions at the same level, justifying the necessity of the multi-stream architectures. (4) Often, there are or blocks heavily connected to many other blocks. (5) Architectures prefer using more than streams, usually using 4 at many levels.
8

---

Published as a conference paper at ICLR 2020
5 CONCLUSION
We present AssembleNet, a new approach for neural architecture search using connection-leamingg guided architecture evolution. AssembleNet finds multi-stream architectures with better connectivity and temporal resolutions for video representation learning. Our experiments confirm that the learned models significantly outperform previous models on two challenging benchmarks.
REFERENCES
Karim Ahmed and Lorenzo Torresani. Connectivity learning multi-branch networks, In Workshop Meta-Learning (MeiaLearn), NeurIPS, 2017.
Gabriel Bender, Pieter-Jan Kindermans, Barret Zoph, Vijay Vasudevan, and Quoe Le. Understanding and simplifying one-shot architecture search. In International Conference on Machine Learning (ICML), 2018.
Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. In Proceedings the IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2017.
Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L. Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE TPAMI, 40(4), 2018.
Ali Diba, Mohsen Fayyaz, Vivek Sharma, Manohar Paluri, Jurgen Gall, Rainer Stiefelhagen, and Lue Van Gool. Holistic large scale video understanding. In 19RR: 1451, 2019.
Christoph Feichtenhofer, Axel Pinz, and Richard Wildes. Spatiotemporal residual networks for video action recognition. In Advances in Neural Information Processing Systems (NeurIPS), pp.
1166--476,, 2016a.
Christoph Feichtenhofer, Axel Pinz, and Andrew Zisserman. Convolutional two-stream network fusion for video action recognition. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition (CVPR), pp. 1933-1941, 2016b.
Christoph Feichtenhofer, Axel Pinz, and Richard P Wildes. Spatiotemporal multiplier networks for video action recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4768 4777,7777700 2017.
Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recognition. arXiv preprint /81.0398/. 2018.
David E. Goldberg and Kalyanmoy Deb. A comparative analysis of selection schemes used in genetic algorithms. In Foundations of Genetic Algorithms, pp. 69-93. Morgan Kaufmann, 1991.
Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
Sergey Toffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning (ICML), 2015.
Colin Lea, Michael D. Flynn, Rene Vidal, Austin Reiter, and Gregory D. Hager. Temporal convolu- tional networks for action segmentation and detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
Chenxi Liu, Barret Zoph, Maxim Neumann, Jonathon Shiens, Wei Hua, Li-Jia Li, Li Fei-Fei. Alan Yuille, Jonathan Huang, and Kevin Murphy. Progressive neural architecture search. In Proceedings of European Conference on Computer Vision (ECCV), 2018.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. DARTS: Differentiable architecture seach. In International Conference Learning Representations (ICLR), 2019.

---

Published as a conference paper at ICLR 2020
Mathew Monfort, Alex Andonian, Bolei Zhou, Kandan Ramakrishnan, Sarah Adel Bargal, Tom Yan, Lisa Brown, Quanfu Fan, Dan Gutfruend, Carl Vondrick, et al. Moments in time dataset: one million videos for event understanding. arXiv preprint arXf::1801.03//55 2018.
AJ Piergiovanni and Michael S Ryoo. Representation flow for action recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
Esteban Real, Alok Aggarwal, Yanping Huang, and Quoe V. Le. Regularized evolution for image classifier architecture search. In Proceedings of AAAI Conference on Artificial Intelligence (AAAI), 2019.
Gunnar A. Sigurdsson, Güil Varol, Xiaolong Wang, Ali Farhadi, Ivan Laptev, and Abhinav Gupta. Hollywood in homes: Crowdsourcing data collection for activity understanding. In Proceedings of European Conference on Computer Vision (ECCV), 2016.
Gunnar A Sigurdsson, Santosh Divvala, Ali Farhadi, and Abhinav Gupta. Asynchronous temporal fields for action recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR),201)
Karen Simonyan and Andrew Zisserman. Two-stream convolutional networks for action recognition in videos. In Advances in Neural Information Processing Systems (NeurIPS), pp. 568-576, 2014.
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE Conference on Computer Vision and Putlern Recognition (CVPR), pp. 281 18-62282, 2016
Du Tran, Lubomir D Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri. C3d: generic features for video analysis. CoRR, abs/1412.0767, 2(7):8, 2014.
Du Tran, Heng Wang, Lorenzo Torresani, Jamie Ray, Yann LeCun, and Manohar Paluri. A closer look at spatiotemporal convolutions for action recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6450-6459, 2018
Limin Wang, Yuanjun Xiong, Zhe Wang. Yu Qiao, Dahua Lin, Xiaoou Tang. and Luc Van Gool. Temporal segment networks: Towards good practices for deep action recognition. Proceedings of European Conference Computer Vision (ECCV), pp. 20-36. Springer, 2016.
Xiaolong Wang and Abhinav Gupta. Videos as space-time region graphs. In Proceedings of European Conference on Computer Vision (ECCV), pp. 399-417, 2018
Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7794-7803, 2018.
Chao- Yuan Wu, Christoph Feichtenhofer, Haoqi Fan, Kaiming He, Philipp Krähenbühh, and Ross Gir- shick. Long-term feature banks for detailed video understanding. arXiv preprint 2018a.
Chao- Yuao Wu, Manzil Zaheer, Hexiang Hu, R Manmatha. Alexander J Smola, and Philipp Krähenbühl. Compressed video action recognition. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp. 6026 60056605,11 2018b.
Saining Xie, Chen Sun, Jonathan Huang, Zhuowen Tu, and Kevin Murphy, Rethinking spatiotemporal feature learning: Speed-accuracy trade-off's in video classification. In Proceedings of European Conference on Computer Vision (ECCV), pp. 305-321, 2018
Saining Xie, Alexander Kirillov, Ross Girshick, and Kaiming He, Exploring randomly wired neural networks for image recognition. In CoRR: 1004.090440000 51569, 2019.
Fisher Yu and Vladlen Koltun. Multi-scale context aggregation by dilated convolutions. In Interna- tional Conference on Learning Representations (ICLR), 2016.
Christopher Zach, Thomas Pock, and Horst Bischof. A duality based approach for realtime tv-1 1 optical flow. In Joint Pattern Recognition Symposium, pp. 214-223. Springer, 2007.
10

---

Published as a conference paper at ICLR 2020
Bolei Zhou, Alex Andonian, Aude Oliva, and Antonio Torralba. Temporal relational reasoning in
videos. In Proceedings of European Conference on Computer Vision (ECCV), pp. 803-818, 2018.
Barret Zoph and Quoc Le. Neural architecture search with reinforcement learning. In International Conference Learning Representations (ICLR), 2017.
Barret Zoph, Vijay Vasudevan, Jonathon Shiens, and Quoe Le. Learning transferable architectures for scalable image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.

---

Published as a conference paper at ICLR 2020
A
APPENDIX
A.1
SUPER-GRAPH VISUALIZATION OF THE CONNECTIVITY SEARCH SPACE
Figure 5 visualizes all possible connections and channel/temporal resolution options our architecture evolution is able to consider. The objective of our evolutionary algorithm could be interpreted as finding the optimal sub-graph (of this super-graph) that maximizes the performance while maintaining the number of total parameters. Trying to directly fit such entire super-graph into the memory was infeasible in our experiments.
C C2RCN0TAA112
Co 1202260
Ce 128256084512
E- 104N
2- 1O4S
ES LONES
E QAAE
EOSLINGTS LOVE
TORANATIO EO LOME
1043
C+
Figure 5: Visualization of the super-graph corresponding to our video architecture search space.
A.2 CHANNEL SIZES OF THE LAYERS AND NODE SPLIT/MERGE MUTATIONS
As we described in the paper, each node (i.e., convolutional block) has parameter C controlling the number of filters of the convolutional layers in the block. When splitting or merging blocks, the number of filters are split or combined respectively. Figure Tigure//provviess a visualization of a block with number of filter specified to the right and split operation. While many designs are possible, design the blocks and splitting as follows. The size of 1xl convolutional layers and ID temporal convolutional layers are strictly governed by C, having the channel size of C (some 4C). On the other hand, the number of 2D convolutional layer is fixed per level as a constant D2 where v is the level of the block. D1 64, D2 = 128, D3 = 256, and D₃ 512. The layers in the stems have 64 channels if there are only two stems and 32 if there are four stems.
When node is split into two nodes, update the resulting two nodes' channel sizes to be half of their original node. This enables us to maintain the total number of model parameters before and after the node split be identical. The first Ixl convolutional layer will have half the parameters after the split, since its output channel size is now 1/2. The 2D convolutional layer will also have exactly half the parameters, since its input channel size is 1/2 while the output channel size is staying fixed. The next 1x1 convolutional layer will have the fixed input channel size while the output channel size becomes 1/2: thus the total number of parameters would be 1/2 of the original parameters.
Merging of the nodes is done an inverse of the way we split. When merging two nodes into one, the merged node inherits all input/output connections from the nodes: we take union of all the connections. The channel size of the merged node the sum of the channel sizes of the two nodes being merged. The temporal dilation rate of the merged node is randomly chosen between the two nodes before the merge.
A.3 HAND-DESIGNED MODELS USED IN THE ABLATION STUDY
Figure 7 illustrates the actual architectures of the hand-designed (2+1)D CNN models used in ablation study. We also show the final learned weights of the connections, illustrating which connections the model ended up using or not using. We note that these architectures are also very enlightening as the connectivity within them are learned in the process. We observe that stronger connections tend to be formed later for 2-stream architectures. For 4-stream architectures, stronger connections do form early, and, not surprisingly, connection to at least node of different
12

---

Published as a conference paper at ICLR 2020
C=128
C=64
C=64
4'C
1x1
1x1
D
20 conv.
conv.
cony.
1D cony.
1D conv.
Node split operation
1x1
1x1
20 conv.
2D
2D
1x1
1x1
Figure 6: An illustration of the node split mutation operator, used for both evolution and initial architecture population generation.
C 256
256 1
51212 r=1
C-512 r=1
C=
C=512
:=1
C 256
C5 r=1
C r-1
C r-1
C 256 r=l
C- C-256 256
C= 256 r-1
C52 r-1
C- 128
C- 128 [=
C- 128
C- 128 r=
C 128
C- 128 r=1
C= r=1
C= 128 [=l
C- 128 t=l
C=128 r=l
C= 128 :=1
C=128 r=l
C=64
C=64
C=64
C= 64
r=l
C= 64 [=1
C=64 [=1
C-64
C-
C=64
C=64
C= c-1
C=61
C=
C=32
C=
C= 32
r=l
t=l
r-1
t=l
[=]
r=l
C= 64 r-1
C-
C 64
C=
C-
C=
C=
e=1
C-64
C= 32 r=
r=4
C=
C=
RGB
Flow
RGB
Flow
Flow
RGB
RGB
Flow
RGB
Flow
(a) Two-stream (late)
(b) Two-stream (fusion Iv. 4)
(c) Two-stream
(d) Two-stream (fully)
(e) Four-streum (fully)
(Flow->RGB)
Figure 7: Illustration hand-designed baseline (2+1)D CNN models used our ablation study.
modality is established, i.e. a node stemming from RGB will connect to at least one flow node at the next level and vice versa.
Below is more detailed description of the networks used in the paper: "Two-stream (late fusion)" means that the model has separate streams every level including the level 4, and the outputs of such two level nodes are combined for the final classification. "Fusion at Iv. 4" is the model that only has one level 4 node combine the outputs of the two level nodes using weighted summation. "Two-stream (fully)" means that the model has two nodes at each level 1-3 and one node level 4, and each node always connected every node in the immediate next level. "Flow-+RGB" means that only the RGB stream nodes combine outputs from both RGB and flow stream nodes of the immediate lower level.
13

---

Published as a conference paper at ICLR 2020
Table 5: The table form of the AssembleNet model with detailed parameters. This model corresponds
to Figure 1. The parameters correspond to {node level, node
node list, C, r, and spatial stride}
Index
Block parameters
0, [RGB], 32,
0, [RGB], 32, 4, 4
0. [Flow]. 32, 1,
0, [Flow] J. 32, 1,
[1]. 32, 1,
[0]. 32, 4,
6
1, 10,1,3, 32, 1,
1,,3,3 32, 2,
3, 64, 2.2 2,
9
10
2. 10, 64,
11
2. 54, 64, 1,2
12
3, [4, 10. 111, 256, 1,2
18, 91, 91, 256, 4,2
14
4, [12, 212,
A.4 ASSEMBLENET MODEL/LAYER DETAILS
We also provide the final AssembleNet model table form in Table In particular, the 2nd element of each block description shows the list of where the input to that block is coming from (i.e., the connections). As already mentioned, 2D and (2+1)D residual modules are repeated in each block. The number of repetitions m are 1.5, 2, 3, and 1.5 at each level. m = 1.5 means that we have one 2D residual module, one (2+ module, and one more 2D module. This makes the number of convolutional layers of each block at levels 1-4 to be 9, 12, 18, and 9. In addition, a stem has at most convolutional layers. The total depth of our network is 50, similar to a conventional (2+1)D ResNet-50. For AssembleNet-101, we use
1.5, 2. 11.5, and 1.5 at each level.
If a block has a spatial stride of 2, the striding happens at the first 2D convolutional layer of the block. In the stem which has the spatial stride of 4, the striding of size 2 happens twice, once at the 2D convolutional layer and at the max pooling layer. As mentioned, the model has a batch normalization layer followed by ReLU after every convolutional layer regardless of its type (i.e., 2D, 1D, and 1x1). 21) conv. filter sizes are 3x3, and 1D conv. filter sizes are 3.
A.5 SINK NODE DETAILS
When each evolved or baseline (2+1)D model is applied to video, it generates 5D (BTYXC) tensor after the final convolutional layer, where B is the size of the batch and is the number of channels. The sink node is responsible for mapping this into the output vector, whose dimensionality is identical to the number of video classes in the dataset. The sink node first applics a spatial average pooling generate 3D (BTC) tensor. If there are multiple level 4 nodes (which rarely is the case), the sink node combines them into single tensor by averaging/concatenatiin them. Averaging or concatenating does not make much difference empirically. Next, temporal average/max pooling is applied to make the representation a 2D (BC) tensor (average pooling was used for the MiT dataset and max pooling was used for Charades). and the final fully connected layer and the soft max layer is applied to generate the final output.
A.6 TRAINING DETAILS
For the Moments in Time (MiT) dataset training. 8 videos are provided per TPU core (with 16GB memory): the total batch size (for each gradient update) is 512 with 32 frames per video. The batch size used for Charades is 128 with 128 frames per video. The base framerate MiT and fps for Charades. The spatial input resolution is 224x224 during training. We used the standard Momentum Optimizer in TensorFlow. We used a learning rate of 3.2 (for MiT) and 25.6 (for Charades), 12k warmup iterations, and cosine decay. No dropout is used, weight decay is set to le-4 and label smoothing set to 0.2.
used is 12.5 fps for
14

---

Published as a conference paper at ICLR 2020
Training model for 10k iterations (during evolution) took 3~-5 hours and fully training the model (for 50k iterations) took ~-24 hours per dataset.
We used the TV-L.1 optical flow extraction algorithm (Zach et al., 2007) implemented with tensor operations by PPergiovannii & Ryoo (2019) to obtain flow input.
A.7 EVALUATION DETAILS
When evaluating model on the MiT dataset, provide 36 frames per video. The duration of each MiT video seconds, making 36 frames roughly correspond to the entire video. For the Charades dataset where each video duration is roughly ~30 seconds, the final class labels are obtained by applying the model to five random 128 frame crops (i.e., segments) of each video. The output multi-class labels are max-pooled get the final label, and is compared to the ground truth to measure the average precision scores. The spatial resolution used for the testing 256x256.
15


Reviews:
[{"review_id": "r1l1KJOAFH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper is a neural architecture search paper. In particular, it applies this to finding better neural architectures for video understanding, emphasizing exploring the video temporal resolutions needed and how to combine intermediate representations capturing appearance and motion. It introduces a somewhat new algorithm for connection-strength-weighted architecture evolution focused on this high-level information fusion problem of video understanding.\n\nI am no expert in the problem domain of this paper (video understanding) but I fond the paper very clear and well-written and easy to understand. The techniques and thinking used seemed good (e.g., using dilated convolutions rather than manual preparing videos with different temporal resolutions!). \n\nThe evolutionary search algorithm was not wildly original or a huge breakthough in the general context of previous work, but seems appropriate, well thought out and works well. \n\nThe results reported are very strong. They get state-of-the-art results on two datasets. I particularly appreciated the evident care in producing strong baselines and ablations (their Charades 2-stream baseline also outperforms all previous work; they show the general strength of four-stream architectures and a random architecture with connection strength learning).\n\nYou mention at the start of section 4.2 that your models have the equivalent number of parameters to ResNet-50. This is good, but you should probably emphasize it more/earlier, since I'd been worrying that you were only winning due to size....", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "S1lGGDPUtB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This submission proposes a way to do multi-stream neural architecture search for video classification. I give an initial rating of accept because (1) there are not many work on video architecture search yet (2) the paper is well written (3) experiments are complete and results are strong. I have a few comments as below. \n\n1. Most work in video action recognition tend not to use optical flow. Many people believe that if 3D conv or (2+1)D conv can be trained well, there is no point in using optical flow. What is the motivation of using flow in this work? I'm interested to know. \n\n2. As shown in Figure 5 of the appendix, the search space is quite large. For each block, it seems that authors search with r=1/2/4/8. However, the best searched network seems to only has r=1, 2 or 4. This is kind of counter-intuitive because longer sequences should in general give better results. Is there an explanation or insight that why r=8 does not show up? \n\n3. The learning rate for both datasets are very high, one is 3.2, the other is 25.6. This is quite unusual. Although many NAS literature show that large learning rate can help to achieve better performance, but 25.6 is really high. Did the authors do learning rate search as well? And what is the search space for learning rate? \n\n4. I understand that this paper focus on learning the connectivity pattern from difference inputs, like rgb and flow. Have the authors tried using RGB alone and searching the architecture? \n\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "HkgiqJWDqB", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nSummary: \n\nThis paper aims to adapt the standard neural architecture search scheme to search a two-input convolutional neural network for video representations. To this end, the paper formulates a direct acyclic graph with two input nodes (for RGB image and optical flow), where each node represents some pre-composed layers and edge represents the data flow with a trainable weight. The searching policy is a modified evolutionary algorithm, which is guided by the trainable weights on the edge, and a set of graph limitations are in-place to avoid over-complicated graphs. The best-selected model outperforms previous baselines and achieves a new state-of-the-art on two video datasets.\n\nOverall, this paper presents a concrete application of neural architecture search for video CNN with interesting results. Edge-weight guided evolutionary algorithm also demonstrates a small improvement in the ablation study. My concern, as detailed later, is if the comparison only with previously human-designed models is necessary. Nevertheless, this paper presents an interesting application of NAS and discover a feasible way to conduct an evolutionary algorithm within a reasonable cost (less than 100 sampled architectures). \n\n\nStrength:\n+ Writing in good shape, easy to follow and understand.\n+ Motivation is clear and timely, reformulate neural architecture search for video representation is novel.\n+ Clear experimental settings and reasonable convincing results.\n\n\nWeakness:\n- Lack of comparison with previous neural architecture search algorithms\nAlthough the results yield that the proposed new search space is meaningful, considering each model has a similar dimension comparing to ResNet-50, it is still unknown if only comparing to human-designed model is a truly fair baseline. In my perspective, since this paper is built on top of NAS strategy with minor adaptation, could the author add one comparison experiment that, the proposed new search space is superior to those previous NAS spaces? For example, one could based on the earlier two-stream ResNet-50 with RGB+F modality, switch the backbone model into a NAS-based one and search with the same evolutionary algorithm (removing the edge weights adaptation). Otherwise, the improvement shown in the paper is not that surprising.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The submission applies architecture search to find effective architectures for video classification. The work is not terribly innovative, but the results are good. All reviewers recommend accepting the paper.
Decision: Accept (Poster)