AssembleNet: Searching for Multi-Stream Neural Connectivity in Video Architectures

1 INTRODUCTION . Learning to represent videos is a challenging problem . Because a video contains spatio-temporal data , its representation is required to abstract both appearance and motion information . This is particularly important for tasks such as activity recognition , as understanding detailed semantic contents of the video is needed . Previously , researchers approached this challenge by designing a two-stream model for appearance and motion information respectively , combining them by late or intermediate fusion to obtain successful results : Simonyan & Zisserman ( 2014 ) ; Feichtenhofer et al . ( 2016b ; a ; 2017 ; 2018 ) . However , combining appearance and motion information is an open problem and the study on how and where different modalities should interchange representations and what temporal aspect/resolution each stream ( or module ) should focus on has been very limited . In this paper , we investigate how to learn feature representations across spatial and motion visual clues . We propose a new multi-stream neural architecture search algorithm with connection learning guided evolution , which focuses on finding higher-level connectivity between network blocks taking multiple input streams at different temporal resolutions . Each block itself is composed of multiple residual modules with space-time convolutional layers , learning spatio-temporal representations . Our architecture learning not only considers the connectivity between such multi-stream , multi-resolution blocks , but also merges and splits network blocks to find better multi-stream video CNN architectures . Our objective is to address two main questions in video representation learning : ( 1 ) what feature representations are needed at each intermediate stage of the network and at which resolution and ( 2 ) how to combine or exchange such intermediate representations ( i.e. , connectivity learning ) . Unlike previous neural architecture search methods for images that focus on finding a good ‘ module ’ of convolutional layers to be repeated in a single-stream networks ( Zoph et al. , 2018 ; Real et al. , 2019 ) , our objective is to search for higher-level connections between multiple sequential or concurrent blocks to form multi-stream architectures . We propose the concept of AssembleNet , a new method of fusing different sub-networks with different input modalities and temporal resolutions . AssembleNet is a general formulation that allows representing various forms of multi-stream CNNs as directed graphs , coupled with an efficient evolutionary algorithm to explore the network connectivity . Specifically , this is done by utilizing the learned connection weights to guide evolution , in addition to randomly combining , splitting , or connecting sub-network blocks . AssembleNet is a ‘ family ’ of learnable architectures ; they provide a generic approach to learn connectivity among feature representations across input modalities , while being optimized for the target task . We believe this is the first work to ( i ) conduct research on automated architecture search with multi-stream connections for video understanding , and ( ii ) introduce the new connection-learning-guided evolutionary algorithm for neural architecture search . Figure 1 shows an example learned AssembleNet . The proposed algorithm for learning video architectures is very effective : it outperforms all prior work and baselines on two very challenging benchmark datasets , and establishes a new state-of-the-art . AssembleNet models use equivalent number of parameters to standard two-stream ( 2+1 ) D ResNet models . 2 PREVIOUS WORK . A video is a spatio-temporal data ( i.e. , image frames concatenated along time axis ) , and its representation must abstract both spatial and temporal information . Full 3D space-time ( i.e. , XYT ) convolutional layers as well as ( 2+1 ) D convolutional layers have been popularly used to represent videos ( Tran et al. , 2014 ; Carreira & Zisserman , 2017 ; Tran et al. , 2018 ; Xie et al. , 2018 ) . Researchers studied replacing 2D convolutional layers in standard image-based CNNs such as Inception ( Szegedy et al. , 2016 ) and ResNet ( He et al. , 2016 ) , so that it can be directly used for video classification . Two-stream network designs , which combine motion and appearance inputs , are commonly used ( e.g. , Simonyan & Zisserman , 2014 ; Feichtenhofer et al. , 2016a ; 2017 ; 2016b ) . Combining appearance information at two different temporal resolutions ( e.g. , 24 vs. 3 frames per second ) with intermediate connections has been proposed by Feichtenhofer et al . ( 2018 ) . Late fusion of the two-stream representations or architectures with more intermediate connections ( Diba et al. , 2019 ) , have also been explored . However , these video CNN architectures are the result of careful manual designs by human experts . Neural Architecture Search ( NAS ) , the concept of automatically finding better architectures based on data , is becoming increasingly popular ( Zoph & Le , 2017 ; Zoph et al. , 2018 ; Liu et al. , 2018 ) . Rather than relying on human expert knowledge to design a CNN model , neural architecture search allows the machines to generate better performing models optimized for the data . The use of reinforcement learning controllers ( Zoph & Le , 2017 ; Zoph et al. , 2018 ) as well as evolutionary algorithms ( Real et al. , 2019 ) have been studied , and they meaningfully outperform handcrafted architectures . Most of these works focus on learning architectures of modules ( i.e. , groupings of layers and their connections ) to be repeated within a fixed single-stream meta-architecture ( e.g. , ResNet ) for image-based object classification . One-shot architecture search to learn differentiable connections ( Bender et al. , 2018 ; Liu et al. , 2019 ) has also been successful for images . However , it is very challenging to directly extend such work to find multi-stream models for videos , as it requires preparing all possible layers and interactions the final architecture may consider using . In multi-stream video CNNs , there are many possible convolutional blocks with different resolutions , and fully connecting them requires a significant amount of memory and training data , which makes it infeasible . Our work is also related to Ahmed & Torresani ( 2017 ) which used learnable gating to connect multiple residual module branches , and to the RandWire network ( Xie et al. , 2019 ) , which showed that randomly connecting a sufficient number of convolutional layers creates performant architectures . However , similar to previous NAS work , the latter focuses only on generating connections between the layers within a block . The meta-architecture is fixed as a single stream model with a single input modality . In this work , our objective is to learn high-level connectivity between multi-stream blocks for video understanding driven by data . We confirm experimentally that in the multi-stream video CNNs , where multiple types of input modalities need to be considered at various resolutions , randomly connecting blocks is insufficient and the proposed architecture learning strategy is necessary . 3 ASSEMBLENET . We propose a new principled way to find better neural architectures for video representation learning . We first expand a video CNN to a multi-resolution , multi-stream model composed of multiple sequential and concurrent neural blocks , and introduce a novel algorithm to search for the optimal connectivity between the blocks for a given task . We model a video CNN architecture as a collection of convolutional blocks ( i.e. , sub-networks ) connected to each other . Each block is composed of a residual module of space-time convolutional layers repeated multiple times , while having its own temporal resolution . The objective of our video architecture search is to automatically ( 1 ) decide the number of parallel blocks ( i.e. , how many streams to have ) at each level of the network , ( 2 ) choose their temporal resolutions , and ( 3 ) find the optimal connectivity between such multi-stream neural blocks across various levels . The highly interconnected convolutional blocks allow learning of the video representations combining multiple input modalities at various temporal resolutions . We introduce the concept of connection-learningguided architecture evolution to enable multi-stream architecture search . We name our final architecture as an ‘ AssembleNet ’ , since it is formulated by assembling ( i.e. , merging , splitting , and connecting ) multiple building blocks . 3.1 GRAPH FORMULATION . In order to make our neural architecture evolution consider multiple different streams with different modalities at different temporal resolutions , we formulate the multi-stream model as a directed acyclic graph . Each node in the graph corresponds to a sub-network composed of multiple convolutional layers ( i.e. , a block ) , and the edges specify the connections between such sub-networks . Each architecture is denoted as Gi = ( Ni , Ei ) where Ni = { n0i , n1i , n2i , · · · } is the set of nodes and Ei is the set of edges defining their connectivity . Nodes . A node in our graph representation is a ResNet block composed of a fixed number of interleaved 2D and ( 2+1 ) D residual modules . A ‘ 2D module ’ is composed of a 1x1 conv . layer , one 2D conv . layer with filter size 3x3 , and one 1x1 convolutional layer . A ‘ ( 2+1 ) D module ’ consists of a temporal 1D convolutional layer ( with filter size 3 ) , a 2D conv . layer , and a 1x1 conv . layer . In each block , we repeat a regular 2D residual module followed by the ( 2+1 ) D residual module m times . Each node has its own block level , which naturally decides the directions of the edges connected to it . Similar to the standard ResNet models , we made the nodes have a total of four block levels ( + the stem level ) . Having multiple nodes of the same level means the architecture has multiple parallel ‘ streams ’ . Figure 1 illustrates an example . Each level has a different m value : 1.5 , 2 , 3 , and 1.5. m = 1.5 means that there is one 2D module , one ( 2+1 ) D module , and one more 2D module . As a result , the depth of our network is 50 conv . layers . We also have a batch normalization layer followed by a ReLU after every conv . layer . There are two special types of nodes with different layer configurations : source nodes and sink nodes . A source node in the graph directly takes the input and applies a small number of convolutional/pooling layers ( it is often referred as the ‘ stem ’ of a CNN model ) . In video CNNs , the input is a 4D tensor ( XYT + channel ) obtained by concatenating either RGB frames or optical flow images along the time axis . Source nodes are treated as level-0 nodes . The source node is composed of one 2D conv . layer of filter size 7x7 , one 1D temporal conv . layer of filter size 5 , and one spatial max pooling layer . The 1D conv . is omitted in optical flow stems . A sink node generates the final output of the model , and it is composed of one pooling , one fully connected , and one softmax layer . The sink node is also responsible for combining the outputs of multiple nodes at the highest level , by concatenating them after the pooling . More details are provided in Appendix . Each node in the graph also has two attributes controlling the convolutional block : its temporal resolution and the number of channels . We use temporally dilated 1D convolution to dynamically change the resolution of the temporal convolutional layers in different blocks , which are discussed more below . The channel size ( i.e. , the number of filters ) of a node could take arbitrary values , but we constrain the sum of the channels of all nodes in the same block level to be a constant so that the capacity of an AssembleNet model is equivalent to a ResNet model with the same number of layers . Temporally Dilated 1D Convolution . One of the objectives is to allow the video architectures to look at multiple possible temporal resolutions . This could be done by preparing actual videos with different temporal resolutions as in Feichtenhofer et al . ( 2018 ) or by using temporally ‘ dilated convolutions as we introduce here . Having dilated filters allow temporal 1D conv . layers to focus on different temporal resolution without losing temporal granularity . This essentially is a 1D temporal version of standard 2D dilated convolutions used in Chen et al . ( 2018 ) or Yu & Koltun ( 2016 ) : Let k be a temporal filter ( i.e. , a vector ) with size 2d + 1 . The dilated convolution operator ∗r is similar to regular convolution but has different steps for the summation , described as : ( F ∗r k ) ( t ) = ∑ t1+rt2=t F ( t1 ) k ( t2 + d ) ( 1 ) where t , t1 , and t2 are time indexes . r indicates the temporal resolution ( or the amount of dilation ) , and the standard 1D temporal convolution is a special case where r = 1 . In the actual implementation , this is done by inserting r − 1 number of zeros between each element of k to generate k′ , and then convolving such zero-inflated filters with the input : F ∗r k = F ∗ k′ . Importantly , the use of the dilated convolution allows different intermediate sub-network blocks ( i.e. , not just input stems ) to focus on very different temporal resolutions at different levels of the convolutional architecture . Note that our temporally dilated convolution is different from the one used in Lea et al . ( 2017 ) , which designed a specific layer to combine representations from different frames with various step sizes . Our layers dilate the temporal filters themselves . Our dilated convolution can be viewed as a direct temporal version of the standard dilated convolutions used in Chen et al . ( 2018 ) ; Yu & Koltun ( 2016 ) . Edges . Each directed edge specifies the connection between two sub-network blocks , and it describes how a representation is transferred from one block to another block . We constrain the direction of each edge so that it is connected from a lower level block to a higher level block to avoid forming a cycle and allow parallel streams . A node may receive inputs from any number of lower-level nodes ( including skip connections ) and provide its output to any number of higher-level nodes . Our architectures use a ( learnable ) weighted summation to aggregate inputs given from multiple connected nodes . That is , an input to a node is computed as F in = ∑ i sigmoid ( wi ) · F outi , where F outi are output tensors ( i.e. , representations ) of the nodes connected to the node and wi are their corresponding weights . Importantly , each wi is considered as a variable that has to be learned from training data through back propagation . This has two key advantages compared to conventional feature map concatenation : ( i ) The input tensor size is consistent regardless of the number of connections . ( ii ) We use learned connection weights to ‘ guide ’ our architecture evolution algorithm in a preferable way , which we discuss more in Section 3.2 . If the inputs from different nodes differ in their spatial size , we add spatial max pooling and striding to match their spatial size . If the inputs have different channel sizes , we add a 1x1 conv . layer to match the bigger channel size . Temporal sizes of the representations is always consistent in our graphs , as there is no temporal striding in our formulation and the layers in the nodes are fully convolutional .

Summary: This paper is a neural architecture search paper. In particular, it applies this to finding better neural architectures for video understanding, emphasizing exploring the video temporal resolutions needed and how to combine intermediate representations capturing appearance and motion. It introduces a somewhat new algorithm for connection-strength-weighted architecture evolution focused on this high-level information fusion problem of video understanding.