Big-Little Net: An Efficient Multi-Scale Feature Representation for Visual and Speech Recognition

1 INTRODUCTION . Deep Convolutional Neural Network ( CNN ) models have achieved substantial performance gains in many computer vision and speech recognition tasks ( He et al. , 2016 ; 2017 ; Vinyals et al. , 2017 ; Sercu & Goel , 2016 ) . However , the accuracy obtained by these models usually grows proportionally with their complexity and computational cost . This poses an issue for deploying these models in applications that require real-time inferencing and low-memory footprint , such as self-driving vehicles , human-machine interaction on mobile devices , and robotics . Motivated by these applications , many methods have been proposed for model compression and acceleration , including techniques such as pruning ( Dong et al. , 2017 ; Li et al. , 2017 ; Han et al. , 2015 ) , quantization ( Hubara et al. , 2016 ; Li & Liu , 2016 ) , and low-rank factorization ( Wen et al. , 2017 ; Ioannou et al. , 2015 ; Zhang et al. , 2016 ) . Most of these methods have been applied to single-scale inputs , without considering multi-resolution processing . More recently , another line of work applies dynamic routing to allocate different workloads in the networks according to image complexity ( Wu et al. , 2018 ; Wang et al. , 2018 ; Figurnov et al. , 2017 ; Veit & Belongie , 2018 ) . Multi-scale feature representations have proven successful for many vision and speech recognition tasks compared to single-scale methods ( Nah et al. , 2017 ; Chen et al. , 2017a ; Tóth , 2017 ; Farabet et al. , 2013 ) ; however , the computational complexity has not been addressed much in multi-scale networks . The computational cost of a CNN model has much to do with the input image size . A model , if running at half of the image size , can gain a remarkable computational saving of 75 % . Based on this fact , we propose an efficient network architecture by combining image information at different scales through a multi-branch network . As shown in Fig . 1 , our key idea is to use a high-complexity branch ( accurate but costly ) for low-scale feature representation and low-complexity branch ( efficient but less accurate ) for high-scale feature representation . The two types of features are frequently merged together to complement and enrich each other , leading to a stronger feature representation than either of them individually . We refer to the deeper branch operating at low image resolution as Big-Branch and the shallower one as Little-Branch to reflect their differences in computation . The new network architecture is thus called Big-Little Net or bL-Net for short in this paper . While being structurally simple , our approach is quite effective . We demonstrate later that when bL-Net is integrated into state-of-the-art CNNs such as ResNet , ResNeXt and SEResNeXt , it yields 2× computational savings over the baselines without losing any accuracy . It also outperforms many other recently developed approaches based on more sophisticated architectures at the same FLOP count . One work that relates to ours is the Inception model ( Szegedy et al. , 2016a ; 2017 ) , which also leverages parallel pathways in a network building block for efficiency . However , the efficiency of these models relies on substantial use of 1×1 and separable filters ( i.e . 1×3 and 3×1 ) . In contrast , bL-Net is general and applicable to many architectures including Inception . The main contributions of our paper are summarized as follows : • We propose an efficient and effective multi-scale CNN architecture for object and speech recognition . • We demonstrate that our approach reduces computation by 1/3 in models such as ResNet and ResNeXt while improving accuracy over 1 % point than the baselines , and the computational savings can be higher up to 1/2 without losing any accuracy ; these results outperform state-of-the-art networks that focus on CNN acceleration by a large margin at the same FLOPs . • We validate the proposed method on a speech recognition task , where we also achieve better word error rates while reducing the number of FLOPs by 30 % . 2 RELATED WORK . Model Compression and Acceleration . Network pruning ( Dong et al. , 2017 ; Li et al. , 2017 ) and quantization ( Hubara et al. , 2016 ; Li & Liu , 2016 ) are popular techniques to remove model redundancy and save computational cost . Another thread of work consists of training a sparse model directly , such as IGCv2 ( Xie et al. , 2018 ) and SCConv ( Fan et al. , 2017 ) . Efficient network architectures like MobileNetV2 ( Sandler et al. , 2018 ) and ShuffleNetV2 ( Ma et al. , 2018 ) have also been explored for training compact deep networks . Other methods include knowledge distillation ( Hinton et al. , 2015 ) , compression with structured matrices ( Cheng et al. , 2015 ; Sindhwani et al. , 2015 ) , and hashing ( Chen et al. , 2015 ) . Dynamic routing ( Wu et al. , 2018 ; Wang et al. , 2018 ; Figurnov et al. , 2017 ; Veit & Belongie , 2018 ) has also been explored in residual networks to improve efficiency . These methods operate with single-resolution inputs , while our approach processes multi-resolution data . It could be used in tandem with these methods to further improve efficiency . Multi-Resolution Feature Representations . The notion of multi-scale feature representation can be dated back to image pyramids ( Adelson et al. , 1984 ) and scale-space theory ( Lindeberg & ter Haar Romeny , 1994 ) . More recently , several methods have proposed multi-scale CNN-based architectures for object detection and recognition . MSCNN ( Cai et al. , 2016 ) , DAG-CNN ( Yang & Ramanan , 2015 ) and FPN ( Lin et al. , 2017 ) use features at different layers to form multi-scale features . Hourglass networks ( Newell et al. , 2016 ) use a hierarchical multi-resolution model for human pose estimation . However , this approach induces a heavy workload as the complexity of each sub-network in their model is equal . Nah et al . ( Nah et al. , 2017 ) and Eigen et al . ( Eigen & Fergus , 2015 ) combine the features from multiple networks working on different resolutions to generate multi-scale features . The overall computational cost grows along with the number of scales , leading to inefficient models . In contrast to existing methods , our approach uses different network capacities for different scales , and yields more powerful multi-scale features by fusing them at multiple levels of the network model . Closely related to our work , approaches such as ( Huang et al. , 2018 ; Saxena & Verbeek , 2016 ) , apply multiple branches at multi-scales while aiming at reducing computational complexity . In contrast to our work , their computational gain mostly comes from early exit depending on the input image . Our approach speeds up a network constantly regardless of the input image . 3 OUR APPROACH . We develop a simple , easy-to-implement , yet very efficient and effective network architecture . It learns multi-scale feature representations by fusing multiple branches with different image scales and computational complexity . As shown in Fig . 1 , we design a multi-scale feature module with the following principles : ( I ) each branch corresponds to a single unique image scale ( or resolution ) ; ( II ) the computational cost of a branch is inversely proportional to the scale . Note that the principle ( II ) implies that we use high-complexity networks at lower resolutions and low-complexity networks at higher resolutions for the sake of efficiency . 3.1 BIG-LITTLE NET . Big-Little Net is a sequence of Big-Little Modules , each one taking input xi and producing output xi+1 . Within a Big-Little Module , assume we have K branches working on K scales [ 1 , 1/2 , 1/4 , . . . , 1/2K−1 ] . We denote a feature map xi at scale 1/2k as xki , indicating the spatial size of xi downsampled by 2k with respect to the original input dimension . We use a weighted sum to combine all branches into a feature representation at scale 1/2k . Mathematically , the module ’ s output xi+1 can be expressed by xi+1 = F ( K−1∑ k=0 ckSk ( fk ( xki ) ) ) , ( 1 ) where fk ( · ) denotes a sequence of convolutional layers . Typically for higher k , fk will have more convolutional layers having more feature maps . Sk ( · ) is the operation that matches the output size of the branches , either : ( 1 ) increasing the number feature maps with a 1× 1 convolution , ( 2 ) upsampling to match the output size of the k = 0 branch , or both . ck indicates the weighting coefficients of each scale in the merge while F ( · ) is an optional final fusion layer like a convolutional layer . Note that branches are merged at the end of every Big-Little Module and merging the branch outputs happens at the highest resolution and highest number of feature maps between the branches . Maintaining these large intermediate states avoids information loss . A crucial aspect of this design is that through consecutive merging and downsampling , the expensive branches operating at low resolution still have access to the high resolution information , processed by the cheaper branches in the previous module . While our design is suitable for any number of networks , in this work we primarily focus on the case of two networks , i.e. , K = 2 . We also experimented with K > 2 in object and speech recognition ; however , the K = 2 case provided the best balance between accuracy and computation ( See Appendix A.4 and Section 4.2 for details ) . Following the principles above , we propose a multinetwork architecture that integrates two branches for multi-scale feature representation . Figure 1 ( b ) shows an example Big-Little Net architecture . The module includes two branches , each of which represents a separate network block from a deep model ( accurate but costly ) and a less deep counterpart ( efficient but less accurate ) . The two branches are fused at the end through linear combination with unit weights ( i.e. , c0 = c1 = 1.0 ) . Before fusion , the low resolution feature maps are upsampled using bilinear interpolation to spatially match the higher-resolution counterparts ( =S1 ( · ) ) . Similarly , the high resolution feature map has an additional 1× 1 convolutional layer to increase the number of output channels ( =S0 ( · ) ) . Furthermore , since our design is based on ResNet , we add a residual block to further fuse the combined features ( i.e. , F ( · ) is a residual block ) . For convenience , we refer to these two branches as Big-Branch ( more layers and channels at low resolution ) and Little-Branch ( fewer layers and channels at high resolution ) , respectively . We also denote the module as Big-Little Module and the entire architecture as Big-Little Net or bL-Net . To control the complexity of bL-Net , we introduce two parameters to specify the complexity of the Little-Branch with respect to the Big-Branch . The Big-Branch typically follows the structure of the original network , but the Little-Branch needs to be heavily slimmed and shortened to reduce computation as it operates on high resolution . Here we use two parameters α and β to control the width and depth of the Little-Branch , respectively . As shown in Fig . 1 ( b ) , α specifies the reduction factor of the number of channels in the convolutional layers of Little-Branch with respect to that of the original network while β is the reduction factor of the number of convolutional layers . Larger values of α and β lead to lower complexity in bL-Net . As demonstrated later , with an appropriate choice of α and β ( See Table 1 and Table 3 ) , the cost of the Little-Branch can be 1/6 of the bL-Net and 1/12 of the original network while still providing sufficient complementary information to the Big-Branch .

Summary: This paper presents a novel multi-scale architecture that achieves a better trade-off speed/accuracy than most of the previous models. The main idea is to decompose a convolution block into multiple resolutions and trade computation for resolution, i.e. low computation for high resolution representations and higher computation for low resolution representations. In this way the low resolution can focus on having more layers and channels, but coarsely, while the high resolution can keep all the image details, but with a smaller representation. The branches (normally two) are merged at the end of each block with linear combination at high resolution. Results for image classification on ImageNet with different network architectures and for speech recognition on Switchboard show the accuracy and speed of the proposed model.