AssembleNet: Searching for Multi-Stream Neural Connectivity in Video Architectures

1 INTRODUCTION . Learning to represent videos is a challenging problem . Because a video contains spatio-temporal data , its representation is required to abstract both appearance and motion information . This is particularly important for tasks such as activity recognition , as understanding detailed semantic contents of the video is needed . Previously , researchers approached this challenge by designing a two-stream model for appearance and motion information respectively , combining them by late or intermediate fusion to obtain successful results : Simonyan & Zisserman ( 2014 ) ; Feichtenhofer et al . ( 2016b ; a ; 2017 ; 2018 ) . However , combining appearance and motion information is an open problem and the study on how and where different modalities should interchange representations and what temporal aspect/resolution each stream ( or module ) should focus on has been very limited . In this paper , we investigate how to learn feature representations across spatial and motion visual clues . We propose a new multi-stream neural architecture search algorithm with connection learning guided evolution , which focuses on finding higher-level connectivity between network blocks taking multiple input streams at different temporal resolutions . Each block itself is composed of multiple residual modules with space-time convolutional layers , learning spatio-temporal representations . Our architecture learning not only considers the connectivity between such multi-stream , multi-resolution blocks , but also merges and splits network blocks to find better multi-stream video CNN architectures . Our objective is to address two main questions in video representation learning : ( 1 ) what feature representations are needed at each intermediate stage of the network and at which resolution and ( 2 ) how to combine or exchange such intermediate representations ( i.e. , connectivity learning ) . Unlike previous neural architecture search methods for images that focus on finding a good ‘ module ’ of convolutional layers to be repeated in a single-stream networks ( Zoph et al. , 2018 ; Real et al. , 2019 ) , our objective is to search for higher-level connections between multiple sequential or concurrent blocks to form multi-stream architectures . We propose the concept of AssembleNet , a new method of fusing different sub-networks with different input modalities and temporal resolutions . AssembleNet is a general formulation that allows representing various forms of multi-stream CNNs as directed graphs , coupled with an efficient evolutionary algorithm to explore the network connectivity . Specifically , this is done by utilizing the learned connection weights to guide evolution , in addition to randomly combining , splitting , or connecting sub-network blocks . AssembleNet is a ‘ family ’ of learnable architectures ; they provide a generic approach to learn connectivity among feature representations across input modalities , while being optimized for the target task . We believe this is the first work to ( i ) conduct research on automated architecture search with multi-stream connections for video understanding , and ( ii ) introduce the new connection-learning-guided evolutionary algorithm for neural architecture search . Figure 1 shows an example learned AssembleNet . The proposed algorithm for learning video architectures is very effective : it outperforms all prior work and baselines on two very challenging benchmark datasets , and establishes a new state-of-the-art . AssembleNet models use equivalent number of parameters to standard two-stream ( 2+1 ) D ResNet models . 2 PREVIOUS WORK . A video is a spatio-temporal data ( i.e. , image frames concatenated along time axis ) , and its representation must abstract both spatial and temporal information . Full 3D space-time ( i.e. , XYT ) convolutional layers as well as ( 2+1 ) D convolutional layers have been popularly used to represent videos ( Tran et al. , 2014 ; Carreira & Zisserman , 2017 ; Tran et al. , 2018 ; Xie et al. , 2018 ) . Researchers studied replacing 2D convolutional layers in standard image-based CNNs such as Inception ( Szegedy et al. , 2016 ) and ResNet ( He et al. , 2016 ) , so that it can be directly used for video classification . Two-stream network designs , which combine motion and appearance inputs , are commonly used ( e.g. , Simonyan & Zisserman , 2014 ; Feichtenhofer et al. , 2016a ; 2017 ; 2016b ) . Combining appearance information at two different temporal resolutions ( e.g. , 24 vs. 3 frames per second ) with intermediate connections has been proposed by Feichtenhofer et al . ( 2018 ) . Late fusion of the two-stream representations or architectures with more intermediate connections ( Diba et al. , 2019 ) , have also been explored . However , these video CNN architectures are the result of careful manual designs by human experts . Neural Architecture Search ( NAS ) , the concept of automatically finding better architectures based on data , is becoming increasingly popular ( Zoph & Le , 2017 ; Zoph et al. , 2018 ; Liu et al. , 2018 ) . Rather than relying on human expert knowledge to design a CNN model , neural architecture search allows the machines to generate better performing models optimized for the data . The use of reinforcement learning controllers ( Zoph & Le , 2017 ; Zoph et al. , 2018 ) as well as evolutionary algorithms ( Real et al. , 2019 ) have been studied , and they meaningfully outperform handcrafted architectures . Most of these works focus on learning architectures of modules ( i.e. , groupings of layers and their connections ) to be repeated within a fixed single-stream meta-architecture ( e.g. , ResNet ) for image-based object classification . One-shot architecture search to learn differentiable connections ( Bender et al. , 2018 ; Liu et al. , 2019 ) has also been successful for images . However , it is very challenging to directly extend such work to find multi-stream models for videos , as it requires preparing all possible layers and interactions the final architecture may consider using . In multi-stream video CNNs , there are many possible convolutional blocks with different resolutions , and fully connecting them requires a significant amount of memory and training data , which makes it infeasible . Our work is also related to Ahmed & Torresani ( 2017 ) which used learnable gating to connect multiple residual module branches , and to the RandWire network ( Xie et al. , 2019 ) , which showed that randomly connecting a sufficient number of convolutional layers creates performant architectures . However , similar to previous NAS work , the latter focuses only on generating connections between the layers within a block . The meta-architecture is fixed as a single stream model with a single input modality . In this work , our objective is to learn high-level connectivity between multi-stream blocks for video understanding driven by data . We confirm experimentally that in the multi-stream video CNNs , where multiple types of input modalities need to be considered at various resolutions , randomly connecting blocks is insufficient and the proposed architecture learning strategy is necessary . 3 ASSEMBLENET . We propose a new principled way to find better neural architectures for video representation learning . We first expand a video CNN to a multi-resolution , multi-stream model composed of multiple sequential and concurrent neural blocks , and introduce a novel algorithm to search for the optimal connectivity between the blocks for a given task . We model a video CNN architecture as a collection of convolutional blocks ( i.e. , sub-networks ) connected to each other . Each block is composed of a residual module of space-time convolutional layers repeated multiple times , while having its own temporal resolution . The objective of our video architecture search is to automatically ( 1 ) decide the number of parallel blocks ( i.e. , how many streams to have ) at each level of the network , ( 2 ) choose their temporal resolutions , and ( 3 ) find the optimal connectivity between such multi-stream neural blocks across various levels . The highly interconnected convolutional blocks allow learning of the video representations combining multiple input modalities at various temporal resolutions . We introduce the concept of connection-learningguided architecture evolution to enable multi-stream architecture search . We name our final architecture as an ‘ AssembleNet ’ , since it is formulated by assembling ( i.e. , merging , splitting , and connecting ) multiple building blocks . 3.1 GRAPH FORMULATION . In order to make our neural architecture evolution consider multiple different streams with different modalities at different temporal resolutions , we formulate the multi-stream model as a directed acyclic graph . Each node in the graph corresponds to a sub-network composed of multiple convolutional layers ( i.e. , a block ) , and the edges specify the connections between such sub-networks . Each architecture is denoted as Gi = ( Ni , Ei ) where Ni = { n0i , n1i , n2i , · · · } is the set of nodes and Ei is the set of edges defining their connectivity . Nodes . A node in our graph representation is a ResNet block composed of a fixed number of interleaved 2D and ( 2+1 ) D residual modules . A ‘ 2D module ’ is composed of a 1x1 conv . layer , one 2D conv . layer with filter size 3x3 , and one 1x1 convolutional layer . A ‘ ( 2+1 ) D module ’ consists of a temporal 1D convolutional layer ( with filter size 3 ) , a 2D conv . layer , and a 1x1 conv . layer . In each block , we repeat a regular 2D residual module followed by the ( 2+1 ) D residual module m times . Each node has its own block level , which naturally decides the directions of the edges connected to it . Similar to the standard ResNet models , we made the nodes have a total of four block levels ( + the stem level ) . Having multiple nodes of the same level means the architecture has multiple parallel ‘ streams ’ . Figure 1 illustrates an example . Each level has a different m value : 1.5 , 2 , 3 , and 1.5. m = 1.5 means that there is one 2D module , one ( 2+1 ) D module , and one more 2D module . As a result , the depth of our network is 50 conv . layers . We also have a batch normalization layer followed by a ReLU after every conv . layer . There are two special types of nodes with different layer configurations : source nodes and sink nodes . A source node in the graph directly takes the input and applies a small number of convolutional/pooling layers ( it is often referred as the ‘ stem ’ of a CNN model ) . In video CNNs , the input is a 4D tensor ( XYT + channel ) obtained by concatenating either RGB frames or optical flow images along the time axis . Source nodes are treated as level-0 nodes . The source node is composed of one 2D conv . layer of filter size 7x7 , one 1D temporal conv . layer of filter size 5 , and one spatial max pooling layer . The 1D conv . is omitted in optical flow stems . A sink node generates the final output of the model , and it is composed of one pooling , one fully connected , and one softmax layer . The sink node is also responsible for combining the outputs of multiple nodes at the highest level , by concatenating them after the pooling . More details are provided in Appendix . Each node in the graph also has two attributes controlling the convolutional block : its temporal resolution and the number of channels . We use temporally dilated 1D convolution to dynamically change the resolution of the temporal convolutional layers in different blocks , which are discussed more below . The channel size ( i.e. , the number of filters ) of a node could take arbitrary values , but we constrain the sum of the channels of all nodes in the same block level to be a constant so that the capacity of an AssembleNet model is equivalent to a ResNet model with the same number of layers . Temporally Dilated 1D Convolution . One of the objectives is to allow the video architectures to look at multiple possible temporal resolutions . This could be done by preparing actual videos with different temporal resolutions as in Feichtenhofer et al . ( 2018 ) or by using temporally ‘ dilated convolutions as we introduce here . Having dilated filters allow temporal 1D conv . layers to focus on different temporal resolution without losing temporal granularity . This essentially is a 1D temporal version of standard 2D dilated convolutions used in Chen et al . ( 2018 ) or Yu & Koltun ( 2016 ) : Let k be a temporal filter ( i.e. , a vector ) with size 2d + 1 . The dilated convolution operator ∗r is similar to regular convolution but has different steps for the summation , described as : ( F ∗r k ) ( t ) = ∑ t1+rt2=t F ( t1 ) k ( t2 + d ) ( 1 ) where t , t1 , and t2 are time indexes . r indicates the temporal resolution ( or the amount of dilation ) , and the standard 1D temporal convolution is a special case where r = 1 . In the actual implementation , this is done by inserting r − 1 number of zeros between each element of k to generate k′ , and then convolving such zero-inflated filters with the input : F ∗r k = F ∗ k′ . Importantly , the use of the dilated convolution allows different intermediate sub-network blocks ( i.e. , not just input stems ) to focus on very different temporal resolutions at different levels of the convolutional architecture . Note that our temporally dilated convolution is different from the one used in Lea et al . ( 2017 ) , which designed a specific layer to combine representations from different frames with various step sizes . Our layers dilate the temporal filters themselves . Our dilated convolution can be viewed as a direct temporal version of the standard dilated convolutions used in Chen et al . ( 2018 ) ; Yu & Koltun ( 2016 ) . Edges . Each directed edge specifies the connection between two sub-network blocks , and it describes how a representation is transferred from one block to another block . We constrain the direction of each edge so that it is connected from a lower level block to a higher level block to avoid forming a cycle and allow parallel streams . A node may receive inputs from any number of lower-level nodes ( including skip connections ) and provide its output to any number of higher-level nodes . Our architectures use a ( learnable ) weighted summation to aggregate inputs given from multiple connected nodes . That is , an input to a node is computed as F in = ∑ i sigmoid ( wi ) · F outi , where F outi are output tensors ( i.e. , representations ) of the nodes connected to the node and wi are their corresponding weights . Importantly , each wi is considered as a variable that has to be learned from training data through back propagation . This has two key advantages compared to conventional feature map concatenation : ( i ) The input tensor size is consistent regardless of the number of connections . ( ii ) We use learned connection weights to ‘ guide ’ our architecture evolution algorithm in a preferable way , which we discuss more in Section 3.2 . If the inputs from different nodes differ in their spatial size , we add spatial max pooling and striding to match their spatial size . If the inputs have different channel sizes , we add a 1x1 conv . layer to match the bigger channel size . Temporal sizes of the representations is always consistent in our graphs , as there is no temporal striding in our formulation and the layers in the nodes are fully convolutional .

Summary: This paper aims to adapt the standard neural architecture search scheme to search a two-input convolutional neural network for video representations. To this end, the paper formulates a direct acyclic graph with two input nodes (for RGB image and optical flow), where each node represents some pre-composed layers and edge represents the data flow with a trainable weight. The searching policy is a modified evolutionary algorithm, which is guided by the trainable weights on the edge, and a set of graph limitations are in-place to avoid over-complicated graphs. The best-selected model outperforms previous baselines and achieves a new state-of-the-art on two video datasets.