Title: BIG-LITTLE NET: AN EFFICIENT MULTI-SCALE FEATURE REPRESENTATION FOR VISUAL AND SPEECH RECOGNITION

Abstract: In this paper, we propose a novel Convolutional Neural Network (CNN) architecture for learning multi-scale feature representations with good tradeoffs between speed and accuracy. This is achieved by using a multi-branch network, which has different computational complexity at different branches with different resolutions. Through frequent merging of features from branches at distinct scales, our model obtains multi-scale features while using less computation. The proposed approach demonstrates improvement of model efficiency and performance on both object recognition and speech recognition tasks, using popular architectures including ResNet, ResNeXt and SEResNeXt. For object recognition, our approach reduces computation by 1/3 while improving accuracy significantly over 1% point than the baselines, and the computational savings can be higher up to 1/2 without compromising the accuracy. Our model also surpasses state-of-the-art CNN acceleration approaches by a large margin in terms of accuracy and FLOPs. On the task of speech recognition, our proposed multi-scale CNNs save 30% FLOPs with slightly better word error rates, showing good generalization across domains.

1. INTRODUCTION:

Deep Convolutional Neural Network (CNN) models have achieved substantial performance gains in many computer vision and speech recognition tasks (He et al., 2016; 2017; Vinyals et al., 2017; Sercu & Goel, 2016) . However, the accuracy obtained by these models usually grows proportionally with their complexity and computational cost. This poses an issue for deploying these models in applications that require real-time inferencing and low-memory footprint, such as self-driving vehicles, human-machine interaction on mobile devices, and robotics.

Motivated by these applications, many methods have been proposed for model compression and acceleration, including techniques such as pruning (Dong et al., 2017; Li et al., 2017; Han et al., 2015) , quantization (Hubara et al., 2016; Li & Liu, 2016) , and low-rank factorization (Wen et al., 2017; Ioannou et al., 2015; Zhang et al., 2016) . Most of these methods have been applied to single-scale inputs, without considering multi-resolution processing. More recently, another line of work applies dynamic routing to allocate different workloads in the networks according to image complexity (Wu et al., 2018; Wang et al., 2018; Figurnov et al., 2017; Veit & Belongie, 2018) . Multi-scale feature representations have proven successful for many vision and speech recognition tasks compared to single-scale methods (Nah et al., 2017; Chen et al., 2017a; Tóth, 2017; Farabet et al., 2013) ; however, the computational complexity has not been addressed much in multi-scale networks.

The computational cost of a CNN model has much to do with the input image size. A model, if running at half of the image size, can gain a remarkable computational saving of 75%. Based on this fact, we propose an efficient network architecture by combining image information at different scales through a multi-branch network. As shown in Fig. 1 , our key idea is to use a high-complexity branch (accurate but costly) for low-scale feature representation and low-complexity branch (efficient but less accurate) for high-scale feature representation. The two types of features are frequently merged together to complement and enrich each other, leading to a stronger feature representation than either of them individually. We refer to the deeper branch operating at low image resolution as Big-Branch and the shallower one as Little-Branch to reflect their differences in computation. The new network architecture is thus called Big-Little Net or bL-Net for short in this paper.

While being structurally simple, our approach is quite effective. We demonstrate later that when bL-Net is integrated into state-of-the-art CNNs such as ResNet, ResNeXt and SEResNeXt, it yields 2× computational savings over the baselines without losing any accuracy. It also outperforms many other recently developed approaches based on more sophisticated architectures at the same FLOP count. One work that relates to ours is the Inception model (Szegedy et al., 2016a; 2017) , which also leverages parallel pathways in a network building block for efficiency. However, the efficiency of these models relies on substantial use of 1×1 and separable filters (i.e. 1×3 and 3×1). In contrast, bL-Net is general and applicable to many architectures including Inception.

The main contributions of our paper are summarized as follows:

• We propose an efficient and effective multi-scale CNN architecture for object and speech recognition.

• We demonstrate that our approach reduces computation by 1/3 in models such as ResNet and ResNeXt while improving accuracy over 1% point than the baselines, and the computational savings can be higher up to 1/2 without losing any accuracy; these results outperform state-of-the-art networks that focus on CNN acceleration by a large margin at the same FLOPs. • We validate the proposed method on a speech recognition task, where we also achieve better word error rates while reducing the number of FLOPs by 30%.

2. RELATED WORK:

Model Compression and Acceleration. Network pruning (Dong et al., 2017; Li et al., 2017) and quantization (Hubara et al., 2016; Li & Liu, 2016) are popular techniques to remove model redundancy and save computational cost. Another thread of work consists of training a sparse model directly, such as IGCv2 (Xie et al., 2018) and SCConv (Fan et al., 2017) . Efficient network architectures like MobileNetV2 (Sandler et al., 2018) and ShuffleNetV2 (Ma et al., 2018) have also been explored for training compact deep networks. Other methods include knowledge distillation (Hinton et al., 2015) , compression with structured matrices (Cheng et al., 2015; Sindhwani et al., 2015) , and hashing (Chen et al., 2015) . Dynamic routing (Wu et al., 2018; Wang et al., 2018; Figurnov et al., 2017; Veit & Belongie, 2018) has also been explored in residual networks to improve efficiency. These methods operate with single-resolution inputs, while our approach processes multi-resolution data. It could be used in tandem with these methods to further improve efficiency.

Multi-Resolution Feature Representations. The notion of multi-scale feature representation can be dated back to image pyramids (Adelson et al., 1984) and scale-space theory (Lindeberg & ter Haar Romeny, 1994) . More recently, several methods have proposed multi-scale CNN-based architectures for object detection and recognition. MSCNN (Cai et al., 2016) , DAG-CNN (Yang & Ramanan, 2015) and FPN (Lin et al., 2017) use features at different layers to form multi-scale features. Hourglass networks (Newell et al., 2016) use a hierarchical multi-resolution model for human pose estimation. However, this approach induces a heavy workload as the complexity of each sub-network in their model is equal. Nah et al. (Nah et al., 2017) and Eigen et al. (Eigen & Fergus, 2015) combine the features from multiple networks working on different resolutions to generate multi-scale features. The overall computational cost grows along with the number of scales, leading to inefficient models. In contrast to existing methods, our approach uses different network capacities for different scales, and yields more powerful multi-scale features by fusing them at multiple levels of the network model.

Closely related to our work, approaches such as (Huang et al., 2018; Saxena & Verbeek, 2016) , apply multiple branches at multi-scales while aiming at reducing computational complexity. In contrast to our work, their computational gain mostly comes from early exit depending on the input image. Our approach speeds up a network constantly regardless of the input image.

3. OUR APPROACH:

We develop a simple, easy-to-implement, yet very efficient and effective network architecture. It learns multi-scale feature representations by fusing multiple branches with different image scales and computational complexity. As shown in Fig. 1 , we design a multi-scale feature module with the following principles: (I) each branch corresponds to a single unique image scale (or resolution); (II) the computational cost of a branch is inversely proportional to the scale. Note that the principle (II) implies that we use high-complexity networks at lower resolutions and low-complexity networks at higher resolutions for the sake of efficiency.

3.1. BIG-LITTLE NET:

Big-Little Net is a sequence of Big-Little Modules, each one taking input x i and producing output x i+1 . Within a Big-Little Module, assume we have K branches working on K scales [1, 1/2, 1/4, . . . , 1/2 K-1 ]. We denote a feature map x i at scale 1/2 k as x k i , indicating the spatial size of x i downsampled by 2 k with respect to the original input dimension. We use a weighted sum to combine all branches into a feature representation at scale 1/2 k . Mathematically, the module's output x i+1 can be expressed by

EQUATION

where f k (•) denotes a sequence of convolutional layers. Typically for higher k, f k will have more convolutional layers having more feature maps. S k (•) is the operation that matches the output size of the branches, either: (1) increasing the number feature maps with a 1 × 1 convolution, (2) upsampling to match the output size of the k = 0 branch, or both. c k indicates the weighting coefficients of each scale in the merge while F (•) is an optional final fusion layer like a convolutional layer.

Note that branches are merged at the end of every Big-Little Module and merging the branch outputs happens at the highest resolution and highest number of feature maps between the branches. Maintaining these large intermediate states avoids information loss. A crucial aspect of this design is that through consecutive merging and downsampling, the expensive branches operating at low resolution still have access to the high resolution information, processed by the cheaper branches in the previous module.

While our design is suitable for any number of networks, in this work we primarily focus on the case of two networks, i.e., K = 2. We also experimented with K > 2 in object and speech recognition; however, the K = 2 case provided the best balance between accuracy and computation (See Appendix A.4 and Section 4.2 for details). Following the principles above, we propose a multinetwork architecture that integrates two branches for multi-scale feature representation. Figure 1 (b) shows an example Big-Little Net architecture.

The module includes two branches, each of which represents a separate network block from a deep model (accurate but costly) and a less deep counterpart (efficient but less accurate). The two branches are fused at the end through linear combination with unit weights (i.e., c 0 = c 1 = 1.0). Before fusion, the low resolution feature maps are upsampled using bilinear interpolation to spatially match the higher-resolution counterparts (=S 1 (•)). Similarly, the high resolution feature map has an additional 1 × 1 convolutional layer to increase the number of output channels (=S 0 (•)). Furthermore, since our design is based on ResNet, we add a residual block to further fuse the combined features (i.e., F (•) is a residual block). For convenience, we refer to these two branches as Big-Branch (more layers and channels at low resolution) and Little-Branch (fewer layers and channels at high resolution), respectively. We also denote the module as Big-Little Module and the entire architecture as Big-Little Net or bL-Net.

To control the complexity of bL-Net, we introduce two parameters to specify the complexity of the Little-Branch with respect to the Big-Branch. The Big-Branch typically follows the structure of the original network, but the Little-Branch needs to be heavily slimmed and shortened to reduce computation as it operates on high resolution. Here we use two parameters α and β to control the width and depth of the Little-Branch, respectively. As shown in Fig. 1 (b), α specifies the reduction factor of the number of channels in the convolutional layers of Little-Branch with respect to that of the original network while β is the reduction factor of the number of convolutional layers. Larger values of α and β lead to lower complexity in bL-Net. As demonstrated later, with an appropriate choice of α and β (See Table 1 and Table 3 ), the cost of the Little-Branch can be 1/6 of the bL-Net and 1/12 of the original network while still providing sufficient complementary information to the Big-Branch.

3.2. NETWORK MERGING:

We consider two options for merging the outputs of the branches. The first option is a linear combination, which joins features from two networks by addition (Newell et al., 2016) . The alternative concatenates the outputs of the two networks along the channel dimension, and if needed, a 1 × 1 convolution can be subsequently applied to reduce the number of feature maps (Szegedy et al., 2015) .

Both merging approaches have their pros and cons. With linear combination, the branches can easily compensate each other, meaning each branch can activate output neurons not activated by the other. However, additional cost is added as both the size of feature maps and the number of channels in the two branches need to be adjusted to be the same before addition. On the other hand, merging by concatenation only needs to spatially align the feature maps. On the other hand, concatenation only needs to align the feature map size, however requires a 1 × 1 convolution reducing the number of channels after concatenation, which is a more expensive operation than the pointwise addition.

While linear combination provides an immediate exchange of the activations of both branches, concatenation relies on the following layers for this exchange. This delay in exchange could possibly be problematic if the information from each branch is destructively altered before merging. For example, a nonlinearity such as ReLU would discard all activations less than zero, effectively ignoring negative features in both branches before merging. Since linear combination does not cause too much overhead and provides better accuracy, we chose linear combination as our merging approach. In Appendix A.4, we empirically show that the linear combination approach performs better than concatenation in object recognition.

4. EXPERIMENTAL RESULTS:

We conducted extensive experiments, as discussed below, to validate the effectiveness of our proposed bL-Net on object and speech recognition tasks. bL-Net can be easily integrated with many modern CNNs and here we chose ResNet (He et al., 2016) as the primary architecture to evaluate our approach. For simplicity, from now on, we denote by bL-M the bL-Net using a backbone network M.

For example, bL-ResNet-50 is the Big-Little net based on ResNet-50.

4.1. OBJECT RECOGNITION:

We used the ImageNet dataset (Russakovsky et al., 2015) for all the experiments below on object recognition. This dataset is a common benchmark for object recognition, which contains 1.28 million training images and 50k validation images with labels from 1000 categories. The details of our experimental setup and the network structures for bL-ResNet-50, 101 and 152 can be found in Appendix A.1. ResNet as the backbone network. We experimented with different complexity control factors (α and β) to better understand their effects on performance. α and β control both the structural and computational complexity of the Little-Branch, which determines the overall computational cost of bL-Net.

As can be seen in Table 1 , all the models based on ResNet-50 yield better performance over the baseline with less computation, clearly demonstrating the advantage of combining low-and highcomplexity networks to balance between speed and accuracy. In addition, the small performance gaps between these models suggest that a computationally light Little-Branch (< 15% of the entire network) can compensate well for the low resolution representation by providing finer image details. We consider α = 2 and β = 4 as the default setting for the following experiments. Furthermore, there are more ablation studies on the design of bL-Net in the Appendix A.4.

We further evaluated our approach on deeper models by using ResNet-101 and ResNet-152 as the backbone networks. We see from Table 2 that bL-ResNet-101 and bL-ResNet-152 behave similarly to bL-ResNet-50. As expected, both of them produce better results against the baseline models and achieving notable computational gains. Interestingly, our approach computationally favors deeper models, as evidenced by the fact that more speedups are observed on bL-ResNet-152 (2.3×) than on bL-ResNet-101 (2.0×) and bL-ResNet-50 (1.4×). This is mainly because the Little Branch operating on low resolution spends less computation in a deeper model.

ResNeXt and SEResNeXt as the backbone network. We extended bL-Net to ResNeXt and SERes-NeXt, two of the more accurate yet compact network architectures. We also experimented with (SE)ResNeXt-50 and (SE)ResNeXt-101 using the 32×4d and 64×4d setting (Xie et al., 2017; Hu et al., 2018) . In our case, the Big-Branch follows the same setting of (SE)ResNeXt; however, we changed the cardinality of the Little-Branch to align with the input channels of a group convolution in the Big-Branch. All the results are shown in Table 2 . bL-ResNeXt-50 achieves a moderate speedup (1.40×) and provides an additional gain of 0.6% in accuracy. However, bL-ResNeXt-101 (32×4d) gains a much more substantial speedup of 2× and seeing 0.12% improvement in accuracy. The same trend can be seen on bL-ResNeXt-101 (64×4d). On the other hand, our bL-SEResNeXts also produce better performance while reducing more FLOPs than SEResNeXts. bL-SEResNeXt-101 achieves 2× speedups and improves accuracy by 0.13%.

Since our bL-Net saves more budget in computation, we can evaluate a model at a larger image scale for better performance, e.g., 256 × 256. As illustrated in Table 2 , our models evaluated at 256 × 256 is consistently better than their corresponding baselines while still using fewer FLOPs. The advantage becomes more pronounced with deeper models. For instance, with 40% reduction on FLOPs, bL-ResNeXt-101@256 (64×4d) boosts the top-1 performance by 1.1%, which is quite impressive given that ResNeXt-101 (64×4d) is a very competitive baseline. Table 2 also shows the running times of bL-Net on GPU, which indicate the practical speedups of these models are consistent with the theoretical FLOP reductions reported in Table 2 .

4.1.1. COMPARISON WITH RELATED WORK:

We first compared our method with the approaches that aim to accelerate ResNets or ResNeXts using techniques such as network pruning and adaptive computations. The results are shown in Figure 2 .

Our bL-Net significantly outperforms all related works regarding FLOPs reduction and accuracy. Our bL-ResNet-101 is ∼ 5% better than the network pruning approaches such as PFEC (Li et al., 2017) and LCCL (Dong et al., 2017) , but still using less computation. When compared to SACT and ACT (Figurnov et al., 2017) , our bL-ResNet-101 improves the accuracy by 5% while using the same number of FLOPs. On the other hand, our bL-ResNet-101 outperforms some of the most recent works including BlockDrop (Wu et al., 2018) , SkipNet (Wang et al., 2018) and ConvNet-AIG (Veit & Belongie, 2018 ) by 3.7%, 2.2%, 1.2% top-1 accuracy at the same FLOPs, respectively. This clearly demonstrates the advantages of a simple fusion of two branches at different scales over the more sophisticated dynamic routing techniques developed in these approaches. In comparison to SPPoint (Kuen et al., 2018) under the same FLOPs, our bL-ResNet-101 surpasses it by 2.2% in accuracy.

We also compared bL-Net with the variants of ResNets, like ResAttNe(X)t (Wang et al., 2017) , SEResNeXt (Hu et al., 2018) and CBAM (Woo et al., 2018) . These models introduces attention mechanisms to enhance feature representations in either the channel or the spatial domain. From Figure 2 , it can be seen that our bL-Net outperforms all of them in both FLOPs and accuracy. Our bL-Net achieves better performance than ResAttNeXt while saving 1.5× FLOPs. It also surpasses ∼1% point with similar FLOPs or the similar accuracy with ∼1.7× FLOPs reduction for both SEResNeXt and CBAM. It's worth noting that bL-Net can be potentially integrated with these models to further improve their accuracy and efficiency.

Finally, we ran a benchmark test on bL-ResNeXt@256 under the PyTorch framework with a batch size of 16 on a K80 GPU, and compared against various models that are publicly available. These models include Inception-V3 (Szegedy et al., 2016a) , Inception-V4 (Szegedy et al., 2017) , Inception-ResNet-V2 (Szegedy et al., 2017) , PolyNet (Zhang et al., 2017) , NASNet (Zoph et al., 2018) , PNASNet (Liu From Fig. 3 , we can see that overall, our bL-ResNeXt gains a better tradeoff between efficiency and accuracy. Compared to the Inception networks, bL-Net are better in both FLOPs and GPU running time. The bL-ResNeXt is also 2% point better than DenseNet at the same running speed, and 2× faster than DualPathNet at the same performance.

NASNet achieves lower FLOPs and higher accuracy than bL-Net; however, the networks result in slow GPU running time since their operations are divided into small pieces (i.e. a fragmented structure), which are not friendly for parallel computations. On the other hand, bL-Net, although requiring more FLOPs, can still enjoy the computation optimization brought by modern GPU cards. Moreover, compared with the lowest FLOP configuration of NASNet, our bL-ResNeXt is 0.5% point better while running 1.5× faster.

To further validate the generality of bL-Net, we also integrated the bL-Net with the highly efficient network, ShuffleNetV2 (Ma et al., 2018) , and the results can be found in Appendix A.5. We also demonstrate the adaptability of bL-Net on the object detection task (see Appendix A.6).

4.2. SPEECH RECOGNITION:

We train ResNet style acoustic models in the hybrid framework on Switchboard+Fisher (2000h) and provide results on Hub5 (Switchboard and Call Home portions). Switchboard is a large dataset with 2000 hours of transcribed speech from 28, 000 speakers, which is actively used as benchmark (Xiong et al., 2016; Saon et al., 2017) akin to ImageNet in the computer vision community. Our ResNet acoustic models are similar to the state of the art models described in (Saon et al., 2017) , though slightly simplified (less fully connected layers) and trained with a simpler procedure (no class balancing). We provide results only after Cross-Entropy training and after decoding with a small language model (4M n-grams). Gains from this setting are typically maintained in the standard further pipelines like fine-tuning with sequence training, using more complex language models.

Appendix B gives a thorough overview of the architecture of the speech acoustic models. The main difference speech acoustic models have compared to image classification networks, is that striding or pooling only happens along the frequency axis, while along in the time direction we need to output dense predictions per frame (Sercu & Goel, 2016) . This means that the branches at different resolutions have a fundamentally different view of the signal as it is propagating through the network; the ratio of resolution in frequency (downsampled in the Big-Branch) vs resolution in time (same between branches) is different. We can think about this as the convolutional kernels having different "aspect ratios" between branches. Therefore we not only expect FLOP reductions in bL-Net, but expect to have increased representational power. In addition, similar to the case in object recognition (Table 2 ), we could process the speech signal at higher frequency resolution than what is computationally feasible for the baseline ResNets.

Table 3 shows the results for the different architectures described in Appendix B. Most results are in line with the observations in the object recognition bL-Net. When comparing the baseline ResNet-22 keeping more feature maps in the full-resolution little branches) is important for performance, even though this increases the FLOPs again. We can summarize the best setting of α = 2 and β = 3 for the little branches at full resolution: make them shorter but with more feature maps. This is consistent with the image classification results. From line 2 vs. line 6, the concatenation merge mode performs similar to the default additive merging, while increasing the number of FLOPs. Line 7 (compare to line 2) shows an experiment with additional branches on the lower layers (See Appendix B). Although there is some gain in WER, the added parameters and compute on the lower layers may not make this a worthwhile trade-off.

4.3. DISCUSSION ON bL-Net:

From the results of both tasks, we observe the following common insights, which enable us to design an efficient multi-scale network with competitive performance: (I) The Little-Branch can be very light-weight, (II) bL-Net performs better when the Little-Branch is wide and shallow (smaller α and larger β), (III) merging is effective when the feature dimension has changed, and (IV) branch merging by addition is more effective than concatenation. (I) is because the Big-Branch can extract essential information, a light Little-Branch is good enough to provide sufficient information the Big-Branch lacks. Regarding (II), wider networks have been shown to perform better than deep networks while using a similar number of parameters. (III) is well-discussed in Appendix A.4. Finally (IV), merging through addition provides better regularization for both branches to learn complementary features to form strong features.

5. CONCLUSION:

We proposed an efficient multi-scale feature representation based on integrating multiple networks for object and speech recognition. The Big-Branches gain significant computational reduction by working at low-resolution input but still extract meaningful features while the Little-Branch enriches the features from high-resolution input but with light computation. On object recognition task, we demonstrated that our approach provides approximately 2× speedup over baselines while improving accuracy, and the result significantly outperforms the state-of-the-art networks by a large margin in terms of accuracy and FLOPs reduction. Furthermore, when using the proposed method on speech recognition task, we gained 0.2% WER and saved 30% FLOPs at the same time. In pratice, the proposed bL-Net shows that the reduced FLOPs can consistently speed up the running time on GPU. That evidence showed that the proposed bL-Net is an efficient multi-scale feature representation structure for competitive performance with less computation. In this paper, we chose ResNet, ResNeXt and SEResNeXt as our backbone networks but bL-Net can be integrated with other advanced network structures, like DenseNet (Huang et al., 2017b) , DualPathNet (Chen et al., 2017b) and NASNet (Zoph et al., 2018) to achieve competitive performance while saving computations. Furthermore, bL-Net can be integrated with those CNN acceleration approaches to make models more compact and efficient. ResBlockB: the first 3 × 3 convolution is with stride 2, and a bi-linear upsampling is applied at the end. ResBlockL: a 1 × 1 convolution is applied at the end to align the channel size. s2: the stride is set to 2 for the 3 × 3 convolution in the ResBlock.

A.2 NETWORK STRUCTURE:

This section shows the details of network structures of our bL-ResNet, and the setting of α and β is 2 and 4, respectively. To understand how do we design bL-ResNet-50 based on ResNet-50, Table 4 shows the details of network structure. We used a bottleneck residual block as a ResBlock, and a ResBlock, C denotes a block composed of 1 × 1, 3 × 3, and 1 × 1 convolutions, where the first 1 × 1 and the 3 × 3 have C/4 kernels and the last 1 × 1 has C kernels.

First, the Big-Branch and the Little-Branch shares a residual block at the transition layer, so the number of residual blocks in each branch will be subtracted by 1. The number of residual blocks in the Little-Branch is defined as L β -1 and at least one, where L is the number of residual blocks in the Big-Branch, and the number of kernels in a convolutional layer would be C α , where C is the number of kernels in the Big-Branch. Thus, for all stages, the number of blocks in the Little-Branch Published as a conference paper at ICLR 2019 is only one, and the number of blocks in the Big-Branch would be the number of blocks in ResNet-50 miuns one.

For bL-ResNet-101 and bL-ResNet-152, we redistributed the residual blocks to different stages to balance the residual blocks at each stage. A ResNet model has 5 stages, and each stage has the same spatial size. These two models accumulate most of the convolutions (or computations) on the 4 th stage, where the size of feature maps is 14 × 14 when input size is 224 × 224. While such a design may be suitable for a very deep model, it likely limits the ability of Big-Branch to learn information at large scales, which mostly resides at earlier stages. Thus, we move some blocks in the 4 th stage of these two models to the 2 nd and 3 rd stages. Table 5 shows the details of bL-ResNet-101 and bLResNet-152. 

× 3(2) 1 × 1, 32 3 × 3, 32 1 × 1, 128 L × 1 1 × 1, 64 3 × 3, 64 1 × 1, 256 B × 4(2) 1 × 1, 32 3 × 3, 32 1 × 1, 128 L × 1 transition layer 28 × 28 1 × 1, 64 3 × 3, 64, s2 1 × 1, 256 × 1 bL-module 28 × 28 1 × 1, 128 3 × 3, 128 1 × 1, 512 B × 7(3) 1 × 1, 64 3 × 3, 64 1 × 1, 256 L × 1 1 × 1, 128 3 × 3, 128 1 × 1, 512 B × 11(7) 1 × 1, 64 3 × 3, 64 1 × 1, 256 L × 2 transition layer 14 × 14 1 × 1, 128 3 × 3, 128, s2 1 × 1, 512 × 1 bL-module 14 × 14 1 × 1, 256 3 × 3, 256 1 × 1, 1024 B × 17(22) 1 × 1, 128 3 × 3, 128 1 × 1, 512 L × 3 1 × 1, 256 3 × 3, 256 1 × 1, 1024 B × 29(35) 1 × 1, 128 3 × 3, 128 1 × 1, 512 L × 6 transition layer 14 × 14 1 × 1, 256 3 × 3, 256 1 × 1, 1024 × 1 ResBlock 7 × 7 1 × 1, 512 3 × 3, 512, s2 1 × 1, 2048 × 3

Average pool 1 × 1 7 × 7 average pooling FC, softmax 1000

For each B block, the first 3 × 3 convolution is with stride 2, and a bi-linear upsampling is applied at the end.

For each L block, a 1 × 1 convolution is applied at the end. s2: the stride is set to 2 for the convolutional layer.

The number in the parethesis denotes the original number of blocks in ResNet.

A.3 PERFORMANCE ON LOW RESOLUTION INPUT:

We analyzed what advantages bL-Net could provide as compared to the network which works on low resolution input directly (ResNet-50-lowres). As shown in Table 6 , ResNet-50-lowres reduces lots of computations but its accuracy is not acceptable; however, bL-ResNet-50 (α = 2 and β = 4) achieves a better balance between accuracy and performance. A similar trend is also observed on a deeper model ResNet-101-lowres. While such performance is unsatisfying compared to the state of the art, it is quite reasonable and expected given that almost 3 ∼ 4× reduction of computation are achieved in such a case.

Figure 4 shows the prediction results from bL-ResNet-50 and ResNet-50-lowres. When both models predict correctly (4 (a) and (b)), the bL-ResNet-50 provides better confidence for the prediction.

Because the object only occupies a small portion of an image, the Little-Branch can still capture the object clearly. On the other hand, when the key features of an object is small, like the shape of beak of a bird (c) and the spots of a ladybug (d), bL-ResNet-50 can easily retain that key feature to predict correctly while ResNet-50-lowres provides wrong predicted label. Nonetheless, if we design the Big-Branch in a way that the output channels is identical to the backbone networks, then the number of kernels in the Big-Branch would be only 1 -α with respect to the total number of kernels of the backbone network; thus, in this case, the overall bL-Net can be more efficient while the performance degradation could be compromised. We compared the performance of these two different merging schemes in Table 7 . Although concatenation approach is more efficient, it performs much worse than addition with a gap of almost 1.5%. This leaves addition as a better choice for bL-Net in both visual and speech tasks.

More-branch in bL-Net As mentioned in Section 3.1, our approach can be extended to a scenario with multiple image scales. We experimented with three scales [1/4, 1/2, 1] on bL-ResNet-50 (K = 3) where ResNet-50 is served as the Big-Branch at the scale of 1/4 of the original input, i.e. 56 × 56.

As indicated in Table 7 , a 3-scale bL-Net requires more FLOPs and parameters due to the fact that the overhead in merging more branches is significant for ResNet-50, but even though, it still cannot provide superior performance of a 2-scale bL-Net. This is because the Big-Branch in the 3-scale bL-Net is downsampled aggressively by 4 times, thus substantially degrade the capability of feature representation in the Big-Branch. In our experiments, we start with an input size of 64 × 49, where 64 is the number of logmel filterbanks, calculated for each utterance on-the-fly. We also stack their first and second derivatives to get 3 input channels, resulting in our final input of dimensionality batch_size × 3 × 64 × 49. Our output is of size batch_size×512×4×1, which is then projected to batch_size×512×1×1 and finally to batch_size × 32k × 1 × 1 for classification. We then perform softmax cross-entropy over this output space of 32k tied CD states from forced alignment, doing phone prediction on the central frame of the input utterance. We report results after Cross-Entropy training, on Hub5'00 (SWB and CH part) after decoding using the standard small 4M n-gram language model with a 30.5k word vocabulary.

All models were trained in PyTorch (Paszke et al., 2017) over 16 epochs on 2 GPUs with per-GPU batch size 256 (total batch size of 512), gradient clipping 10.0, weight decay 1 × 10 -6 , nesterov accelerated momentum 0.9, and learning rate 0.03 (annealed by √ 0.5 per epoch ≥ 10).

B.2 NETWORK STRUCTURES:

ResNet-22 Our models follow a ResNet architecture without padding in time (Saon et al., 2017) , which accounts for the fact that padding in time adds undesirable artifacts when processing a longer utterance (Sercu & Goel, 2016) . Under this constraint, each convolution operation reduces our input sequence in time by k -1, where k is the kernel width used. This effect can be seen in Table 12 , in which the time variable, T , is reduced in accordance with the number of convolutions. For similar reasons, when we stride we only do so in frequency, and not in time. For the rest of this section, when we refer to striding we are referring only to striding in frequency. We define our residual blocks as a series of 3 × 3 convolutions. When we transition from one stage to the next, we stride by 2 on the first convolution of the following block. All of our models start with a 5 × 5 convolution with stride 2, which downsamples the input from 64 melbins to 32. bL-ResNet-22 (α = 4, β = 1) Table 12 displays two bL-Net architectures that we experimented with based on the ResNet-22 baseline. The bL-Net baseline, bL-ResNet-22 (α = 4, β = 1), consists of two branches in each Big-Little Module and is well-defined through the parameters α and β. In between each Big-Little Module we downsample our input using a transition layer consisting of a residual block with a single 3 × 3 convolution with stride 2. Therefore, we shift one convolution operation out of the last residual block in each stage that precedes a transition layer.

Whenever downsampling is performed in the Big-Branch, it is only in the frequency dimension and not in time. Similarly, bilinear upsampling only occurs in the frequency dimension. All bL-Net variants end with the same projection and output layer as ResNet-22. All merges, unless otherwise specified, are through linear combination with unit weights per branch. For comparison, we experimented with a version of bL-ResNet-22 (α = 4, β = 1) using concatenation to merge branches, the results of of the network. In the last stage, however, we opt to not branch and instead follow identically the final stage of ResNet-22 with two residual blocks operating at the full input resolution with 512 channels.

bL-PYR-ResNet-22 (α = 4, β = 1) We additionally present results on a pyramidal structure, in which the first stage of the network operates with four branches, the second with three, the third with two, and the fourth equivalent to the fourth stage of ResNet-22 (and bL-ResNet-22 (α = 2, β = 3)).

Due to the setting of α = 4, we increased the number of channels in the first stage Big-Branch to have 256 channels (with the three Little-Branches in this stage having 64, 16, and 4 channels), avoiding a single channel on the smallest branch. Note that the middle branches require both resolution upsampling and 1 × 1 convolution to match channels. The third stage Big-Little Module operates on two branches and is identical to the analagous stage in bL-ResNet-22 (α = 4, β = 1).

:



Figures and Tables:

Figure 1: Our proposed Big-Little Net (bL-Net) for efficient multi-scale feature representations. (a) The bL-Net stacks several Big-Little Modules. A bL-module include K branches (K = 2 in this illustration) where the k th branch represents an image scale of 1/2 k . 'M' here denotes a merging operation. (b) Our implementation of the Big-Little Module includes two branches. The Big-Branch has the same structure as the baseline model while the Little-Branch reduces the convolutional layers and feature maps by α and β, respectively. Larger values of α and β lead to lower computational complexity in Big-Little Net.

Figure 2: Comparison with the ResNet and ResNeXt related works.

Figure 3: Comparison performance among other types of networks. (a) FLOPs. (b) GPU Speed. et al., 2018), DualPathNet(Chen et al., 2017b)  and DenseNet(Huang et al., 2017b). Among them, NASNet currently achieves the best accuracy on ImageNet.

Figure4: Prediction results for bL-ResNet-50 and ResNet-50-lowres. True labels, predicted labels and their probability are listed in the table. When both models predicts correctly ((a) and (b)), bL-ResNet-50 achieves much higher probability; on the other hand, bL-ResNet-50 captures the details on the object and then predicts correctly ((c) and (d)).

Our baseline model, ResNet-22, consists of four stages of two-convolution residual blocks: (3 × 3, 64)×3; (3×3, 128, s2)×3; (3×3, 256, s2)×3; (3×3, 512, s2)×2. The output then goes through a bottleneck projection layer to the 32k-dimensional output CD state: (4 × 1, 512) and (1 × 1, 32k).

Complexity study of the Little-Branch (α and β) for bL-ResNet-50.

Table content: <table><tr><td>Model</td><td colspan="3">Top-1 Error FLOPs (10 9 ) Params (10 6 )</td></tr><tr><td>ResNet-50</td><td>23.66%</td><td>4.09</td><td>25.55</td></tr><tr><td>bL-ResNet-50 (α = 2, β = 2)</td><td>22.72%</td><td>2.91 (1.41×)</td><td>26.97</td></tr><tr><td>bL-ResNet-50 (α = 2, β = 4)</td><td>22.69%</td><td>2.85 (1.44×)</td><td>26.69</td></tr><tr><td>bL-ResNet-50 (α = 4, β = 2)</td><td>23.20%</td><td>2.49 (1.64×)</td><td>26.31</td></tr><tr><td>bL-ResNet-50 (α = 4, β = 4)</td><td>23.15%</td><td>2.48 (1.65×)</td><td>26.24</td></tr></table>

Performance comparison for bL-ResNet, bL-ResNeXt and bL-SEResNeXt. We trained all the ResNet, ResNeXt and SEResNeXt models by ourselves, so the accuracy is slightly different from the papers.

Table content: <table><tr><td>Model</td><td colspan="4">Top-1 Error FLOPs (10 9 ) Params (10 6 ) Speed (ms/batch)  †</td></tr><tr><td>ResNet-101</td><td>21.95%</td><td>7.80</td><td>44.54</td><td>186</td></tr><tr><td>bL-ResNet-101 (α = 2, β = 4)</td><td>21.80%</td><td>3.89 (2.01×)</td><td>41.85</td><td>140 (1.33×)</td></tr><tr><td>bL-ResNet-101@256 (α = 2, β = 4)</td><td>21.04%</td><td>5.08 (1.54×)</td><td>41.85</td><td>162 (1.15×)</td></tr><tr><td>ResNet-152</td><td>21.51%</td><td>11.51</td><td>60.19</td><td>266</td></tr><tr><td>bL-ResNet-152 (α = 2, β = 4)</td><td>21.16%</td><td>5.04 (2.28×)</td><td>57.36</td><td>178 (1.49×)</td></tr><tr><td>bL-ResNet-152@256 (α = 2, β = 4)</td><td>20.34%</td><td>6.58 (1.75×)</td><td>57.36</td><td>205 (1.30×)</td></tr><tr><td>ResNeXt-50 (32×4d)</td><td>22.20%</td><td>4.23</td><td>25.03</td><td>157</td></tr><tr><td>bL-ResNeXt-50 (32×4d) (α = 2, β = 4)</td><td>21.60%</td><td>3.03 (1.40×)</td><td>26.19</td><td>125 (1.26×)</td></tr><tr><td>bL-ResNeXt-50@256 (α = 2, β = 4)</td><td>20.96%</td><td>3.95 (1.08×)</td><td>26.19</td><td>153 (1.03×)</td></tr><tr><td>ResNeXt-101 (32×4d)</td><td>21.20%</td><td>7.97</td><td>44.17</td><td>269</td></tr><tr><td>bL-ResNeXt-101 (32×4d) (α = 2, β = 4)</td><td>21.08%</td><td>4.08 (1.95×)</td><td>41.51</td><td>169 (1.59×)</td></tr><tr><td>bL-ResNeXt-101@256 (α = 2, β = 4)</td><td>20.48%</td><td>5.33 (1.50×)</td><td>41.51</td><td>203 (1.33×)</td></tr><tr><td>ResNeXt-101 (64×4d)</td><td>20.73%</td><td>15.46</td><td>83.46</td><td>485</td></tr><tr><td>bL-ResNeXt-101 (64×4d) (α = 2, β = 4)</td><td>20.48%</td><td>7.14 (2.17×)</td><td>77.36</td><td>263 (1.98×)</td></tr><tr><td>bL-ResNeXt-101@256 (64×4d) (α = 2, β = 4)</td><td>19.65%</td><td>9.32 (1.66×)</td><td>77.36</td><td>318 (1.53×)</td></tr><tr><td>SEResNeXt-50 (32×4d)</td><td>21.78%</td><td>4.23</td><td>27.56</td><td>216</td></tr><tr><td>bL-SEResNeXt-50 (32×4d) (α = 2, β = 4)</td><td>21.44%</td><td>3.03 (1.40×)</td><td>28.77</td><td>163 (1.33×)</td></tr><tr><td>bL-SEResNeXt-50@256 (32×4d) (α = 2, β = 4)</td><td>20.74%</td><td>3.95 (1.08×)</td><td>28.77</td><td>192 (1.03×)</td></tr><tr><td>SEResNeXt-101 (32×4d)</td><td>21.00%</td><td>7.97</td><td>48.96</td><td>376</td></tr><tr><td>bL-SEResNeXt-101 (32×4d) (α = 2, β = 4)</td><td>20.87%</td><td>4.08 (1.95×)</td><td>45.88</td><td>235 (1.60×)</td></tr><tr><td>bL-SEResNeXt-101@256 (32×4d) (α = 2, β = 4)</td><td>19.87%</td><td>5.33 (1.50×)</td><td>45.88</td><td>270 (1.39×)</td></tr><tr><td/><td/><td/><td/><td/></tr></table>

Speech recognition results. We present results on Hub5 and the CallHome portion of Hub5, while the RT-02 Switchboard set was used for selecting decode epoch and HMM prior settings. ) to the best bL-ResNet-22 (line 5), we see not only a reduction in FLOPs, but also a modest gain in Word Error Rate (WER). Comparing lines 2-4, we see that increasing β (i.e. shorter little branches at full resolution) causes no WER degradation, while reducing the number of FLOPs. From line 5 we see that, similar to the object recognition ResNet results, decreasing α from 4 to 2 (i.e.

Table content: <table><tr><td>Model</td><td colspan="4">FLOPs (10 9 ) Params (10 6 ) WER Avg Hub5</td><td>Hub5 CH</td></tr><tr><td>1 Baseline: ResNet-22</td><td>1.11</td><td>3.02</td><td>14.67%</td><td colspan="2">11.15% 18.17%</td></tr><tr><td>2 bL-ResNet-22 (α = 4, β = 1)</td><td colspan="2">0.68 (1.63×) 3.15</td><td>14.72%</td><td colspan="2">11.24% 18.18%</td></tr><tr><td>3 bL-ResNet-22 (α = 4, β = 2)</td><td colspan="2">0.66 (1.68×) 3.11</td><td>14.47%</td><td colspan="2">10.95% 17.95%</td></tr><tr><td>4 bL-ResNet-22 (α = 4, β = 3)</td><td colspan="2">0.65 (1.70×) 3.10</td><td>14.66%</td><td colspan="2">11.25% 18.05%</td></tr><tr><td>5 bL-ResNet-22 (α = 2, β = 3)</td><td colspan="2">0.77 (1.43×) 3.07</td><td>14.46%</td><td colspan="2">11.10% 17.80%</td></tr><tr><td>6 bL-ResNet-22 (α = 4, β = 1) cat</td><td colspan="2">0.70 (1.58×) 3.18</td><td>14.67%</td><td colspan="2">11.31% 18.00%</td></tr><tr><td colspan="3">7 bL-PYR-ResNet-22 (α = 4, β = 1) 0.98 (1.13×) 3.32</td><td>14.50%</td><td colspan="2">11.05% 17.92%</td></tr><tr><td>(line 1</td><td/><td/><td/><td/></tr></table>

Network configurations of bL-ResNet-50. Output size is illustrated in the parenthesis.

Table content: <table><tr><td>Layers</td><td colspan="2">bL-ResNet-50</td><td/><td>ResNet-50</td></tr><tr><td>Convolution</td><td/><td colspan="2">7 × 7, 64, s2 (112 × 112)</td></tr><tr><td/><td/><td>3×3, 32</td><td/></tr><tr><td>bL-module</td><td>3 × 3, 64, s2</td><td>3×3, 32, s2</td><td colspan="2">(56 × 56) MaxPooling (56 × 56)</td></tr><tr><td/><td/><td>1×1, 64</td><td/></tr><tr><td>bL-module</td><td>ResBlock B , 256 ×2</td><td colspan="2">ResBlock L , 128 ×1</td><td>ResBlock, 256 ×3, s1</td></tr><tr><td/><td colspan="2">ResBlock, 256, s2 (28 × 28)</td><td/><td>(56 × 56)</td></tr><tr><td>bL-module</td><td>ResBlock B , 512 ×3</td><td colspan="2">ResBlock L , 256 ×1</td><td>ResBlock, 512 ×4, s2</td></tr><tr><td/><td colspan="2">ResBlock, 512, s2 (14 × 14)</td><td/><td>(28 × 28)</td></tr><tr><td>bL-module</td><td>ResBlock B , 1024 ×5</td><td colspan="2">ResBlock L , 512 ×1</td><td>ResBlock, 1024 ×6, s2</td></tr><tr><td/><td colspan="2">ResBlock, 1024 (14 × 14)</td><td/><td>(14 × 14)</td></tr><tr><td>ResBlock</td><td/><td colspan="3">ResBlock, 2048 ×3, s2 (7 × 7)</td></tr><tr><td>Average pool</td><td/><td colspan="2">7 × 7 average pooling</td></tr><tr><td>FC, softmax</td><td/><td>1000</td><td/></tr></table>

Network configurations of bL-ResNets, and α = 2 and β = 4.

Table content: <table><tr><td>Layers</td><td>Output Size</td><td>bL-ResNet-101</td><td>bL-ResNet-152</td></tr><tr><td>Convolution</td><td>112 × 112</td><td>7 × 7, 64, s2</td><td/></tr><tr><td/><td/><td/><td>3×3, 32</td></tr><tr><td>bL-module</td><td>56 × 56</td><td>3 × 3, 64, s2</td><td>3×3, 32, s2</td></tr><tr><td/><td/><td/><td>1×1, 64</td></tr><tr><td/><td/><td>1 × 1, 64</td><td/></tr><tr><td>bL-module</td><td>56 × 56</td><td>3 × 3, 64</td><td/></tr><tr><td/><td/><td>1 × 1, 256 B</td><td/></tr></table>

Performance of ResNets at different input resolutions.

Table content: <table><tr><td>Network</td><td colspan="3">Top-1 Error FLOPs (10 9 ) Params (10 6 )</td></tr><tr><td>ResNet-50</td><td>23.66%</td><td>4.09</td><td>25.55</td></tr><tr><td>ResNet-50-lowres</td><td>26.10%</td><td>1.29 (3.17×)</td><td>25.60</td></tr><tr><td>ResNet-101</td><td>21.95%</td><td>7.80</td><td>44.54</td></tr><tr><td>ResNet-101-lowres</td><td>24.80%</td><td>2.22 (3.51×)</td><td>44.57</td></tr><tr><td>A.4 ABLATION STUDY</td><td/><td/><td/></tr></table>

Different scales and merging schemes on bL-ResNet.

Table content: <table><tr><td>Network</td><td colspan="3">Top-1 Error FLOPs (10 9 ) Params (10 6 )</td></tr><tr><td>ResNet-50</td><td>23.66%</td><td>4.09</td><td>25.55</td></tr><tr><td>bL-ResNet-50 (addition, K = 2)</td><td>22.69%</td><td>2.85 (1.43×)</td><td>26.69</td></tr><tr><td>bL-ResNet-50 (concatenation, K = 2)</td><td>24.04%</td><td>2.01 (2.03×)</td><td>20.57</td></tr><tr><td>bL-ResNet-50 (addition, K = 3)</td><td>24.12%</td><td>3.91 (1.04×)</td><td>27.23</td></tr></table>

Complexity study of the Little-Branch (α and β) for bL-ResNet-50 and bL-ResNet-101.

Table content: <table><tr><td>Model</td><td colspan="3">Top-1 Error FLOPs (10 9 ) Params (10 6 )</td></tr><tr><td>ResNet-50</td><td>23.66%</td><td>4.09</td><td>25.55</td></tr><tr><td>bL-ResNet-50 (α = 1, β = 1)</td><td>21.75%</td><td>5.65</td><td>34.12</td></tr><tr><td>bL-ResNet-50 (α = 1, β = 2)</td><td>22.11%</td><td>4.34</td><td>30.14</td></tr><tr><td>bL-ResNet-50 (α = 2, β = 2)</td><td>22.72%</td><td>2.91</td><td>26.97</td></tr><tr><td>bL-ResNet-50 (α = 2, β = 4)</td><td>22.69%</td><td>2.85</td><td>26.69</td></tr><tr><td>bL-ResNet-50 (α = 4, β = 2)</td><td>23.20%</td><td>2.49</td><td>26.31</td></tr><tr><td>bL-ResNet-50 (α = 4, β = 4)</td><td>23.15%</td><td>2.48</td><td>26.24</td></tr><tr><td>ResNet-101</td><td>21.95%</td><td>7.80</td><td>44.54</td></tr><tr><td>bL-ResNet-101 (α = 1, β = 1)</td><td>20.31%</td><td>10.29</td><td>63.32</td></tr><tr><td>bL-ResNet-101 (α = 2, β = 2)</td><td>21.40%</td><td>4.27</td><td>43.39</td></tr><tr><td>bL-ResNet-101 (α = 2, β = 4)</td><td>21.80%</td><td>3.89</td><td>41.85</td></tr><tr><td colspan="2">B bL-Net FOR SPEECH RECOGNITION</td><td/><td/></tr><tr><td>B.1 EXPERIMENTAL SETUP</td><td/><td/><td/></tr></table>

Peer review (iclr2019):
 The authors propose a new CNN architecture and show results on object and speech recognition. In particular, they propose a multi-scale CNN module that processes feature maps at various scales. They show compelling results on IN and a reduction of compute complexity Pros: (+) The paper is well written (+) The method is elegant and reproducible (+) Results are compelling and experimentation is thorough Cons: (-) Transfer to other visual tasks, beyond IN, is missing (-) Memory requirements are not mentioned, besides FLOPs, speed and parameters Overall, the proposed approach is elegant and clear. The impact of the multi-scale module is evident, in terms of FLOPs and performance. While their approach performs a little worse than NASNet, both in terms of FLOP efficiency and top1-error, it is simpler and easier to train. I*d like for the authors to also discuss memory requirements for training and testing the network. Finally, various papers have appeared over the recent years showing improvements over baselines on ImageNet. However, most of these papers are not impactful, because they do not show any impact to other visual tasks, such as detection. On the contrary, methods that do transfer get adopted very fast. I would be much more convinced of this approach, if the authors showed similar performance gains (both in terms of complexity and metrics) for COCO detection.