An Empirical Study of Adder Neural Networks for Object Detection

1 Introduction . Object detection is a foundational problem in computer vision and has attracted tremendous interests from both academic and industrial communities for decades [ 21 ] . It has a wide range of applications for various areas , e.g. , video surveillance , autonomous driving and robotic vision . Deep neural networks have indeed dominated the research of object detection in recent years since the pioneering work of R-CNN [ 12 ] . The performance of object detectors has been considerably improved by various designs of architecture , loss function etc . However , most modern accurate object detectors require massive computation , making them quite challenging in resource-constraint applications , e.g. , mobile phones and embedded devices . Various approaches have been proposed to compress and accelerate convolutional neural networks ( CNNs ) for classification tasks , including channel pruning [ 24 , 14 , 26 , 38 ] , low-bit quantization [ 50 , 25 ] and lightweight network design [ 33 , 27 ] . These methods reduce the number of parameters or inference latency while maintaining the accuracy to the maximum extent . Such model compression methods have also been explored in a variety of down-stream tasks , such as semantic segmentation [ 7 ] and image super resolution [ 44 ] etc . There have been a few methods aiming at fast and efficient object detectors . One family of solutions is using new architecture design [ 23 , 31 , 29 , 41 ] . For example , YOLO series [ 30 , 31 ] have achieved good trade-off between running speed and accuracy via a novel one-stage detection framework . Another family of solutions is to use common model compression methods for accelerating object ∗Corresponding author . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . detectors , e.g. , knowledge distillation [ 4 , 42 ] and pruning [ 1 ] . Moreover , some recent works utilize neural architecture search ( NAS ) approach for searching better architectures for different components of object detectors [ 3 , 13 , 11 ] . Although these methods mentioned above show strong performance while improving the efficiency , they are mainly built with traditional convolutional neural networks , which contain massive inefficient multiplications . Recently , Chen et al . [ 5 ] proposed the adder neural networks ( AdderNets ) to replace traditional convolutional filters with adder filters . Since addition is more energy efficient than multiplication [ 36 , 47 ] , AdderNets shed light on the design of efficient neural networks and have the potential of much fewer chip areas and less energy consumption . AdderNets have shown impressive performance in large scale image classification [ 45 ] via a kernel-based progressive distillation method , and also successfully been applied for other applications like image super resolution . Song et al . [ 34 ] proposed to utilize self shortcuts and learnable power activations to build super resolution networks via adder filters . Existing variants of AdderNets mainly deal with either image classification [ 5 , 45 , 9 ] or superresolution tasks [ 34 ] . It is not clear yet how will the AdderNets perform for object detection , which often has sophisticated framework design and various objectives . This much more challenging computer vision task therefore brings in new challenges and opportunities for the research on AdderNets . So how to build accurate and efficient object detectors via AdderNets ? The straightforward idea is to directly replace the original convolutional filters by adder filters . However , this naive adder detector can not be easily trained as classification networks . At first , most modern detectors tend to pre-train the backbone on the ImageNet and then fine-tune the whole model on the target dataset , but this straightforward fine-tuning might worsen adder detectors , because of the sensitivity of adder filter . Moreover , the performance-proven neural architectures were all developed for convolution based detectors , and whether they are still applicable for the adder detectors is unclear . In this paper , we propose a series of strategies to reform efficient object detectors with adder filters . In contrast with the frozen batch normalization widely exploited during fine-tuning of the detector , we empirically observe an opposite conclusion that adder detectors are better to unlock the statistics of batch normalization in pre-trained adder backbone for a performance improvement . Extensive ablation studies are conducted to explore the properties of batch normalization layers and the impact of batch size . In addition , a new feature fusion network with more residual connections and a better fusion module is explored to compensate for sparse adder features . Experimental results are reported on PASCAL VOC and COCO benchmarks , and the results are carefully analyzed and discussed . In particular , the proposed Adder FCOS achieves a 37.8 % mAP on COCO val set , which is comparable with state-of-the-art object detectors , while saving much energy consumption as shown in Figure 1 . In summary , we present an extensive empirical study for how to build object detectors via adder neural networks . We believe that the discussions and analysis in this paper will be beneficial for the research of efficient object detection and adder neural networks . 2 Related Work . Object Detection . Object detection has attracted tremendous interests for decades . Although there have been huge improvements for accurate object detection [ 21 ] , the struggle for high-speed and energy-efficient detectors is largely unsolved . Several recent approaches exploit neural architecture search ( NAS ) to discover efficient and accurate object detectors [ 3 , 13 , 11 ] . On the meanwhile , there are some attempts to utilize model compression methods for improving the efficiency of detectors , e.g. , channel pruning [ 1 ] , quantization [ 43 ] and knowledge distillation [ 4 , 42 ] . Nevertheless , convolutional neural networks ( CNNs ) have in fact dominated the design of object detectors , which consist of massive multiplications and are quite energy inefficient . Adder Neural Networks . Chen et al . [ 5 ] provided a new perspective for designing energy efficient neural networks . They proposed to utilize ` 1 distance to measure the similarity between the input features and filter weights . The devised AdderNets consist of only additions instead of massive multiplication , which are potentially energy efficient . Full precision gradients along with adaptive local learning rate are exploited to optimize this novel neural networks . AdderNets show impressive performance on large-scale image classification benchmark , i.e. , ImageNet . To further improve the performance of AdderNets , Xu et al . [ 45 ] proposed a kernel based knowledge distillation method to learn from homogeneous CNNs . Using this method , AdderNets achieve even better accuracy than ResNet-50 on ImageNet . You et al . [ 47 ] further proposed to use bit-shift operations and additions to improve the energy efficiency . Song et al . [ 34 ] analyzed several key problems of AdderNets and successfully applied AdderNets for image super-resolution . Nevertheless , designing accurate object detectors with AdderNets is still less explored and is the main focus of this paper . 3 AdderNets for Object Detection . Traditional deep convolutional neural networks are mainly constructed by convolutional filters , which have massive manipulations and are energy inefficient . To this end , Chen et al . [ 5 ] proposed a new kind of neural architecture called AdderNet , which adopts ` 1-norm to compute the similarity between the input features X and the filter weights F , as shown in Eq . ( 1 ) . Y ( m , n , t ) = − d∑ i=0 d∑ j=0 cin∑ k=0 |X ( m+ i , n+ j , k ) − F ( i , j , k , t ) | . ( 1 ) We aim to extend the success of AdderNet to the object detection task . We build our adder object detectors upon modern convolution-based detection frameworks , e.g. , FCOS [ 39 ] . It is straightforward to replace the convolutional filters in the detector with adder filters . However , it is non-trivial to train an adder detector of decent performance . We first analyze several key strategies for applying adder filters for detectors . We then propose a novel multi-scale feature fusion architecture which is more suitable for adder detectors . 3.1 Making it Work : Towards a Strong Baseline . Intuitively if we replace the convolutional filters with adder filters in modern object detectors like FCOS [ 39 ] , we could obtain a vanilla adder detector , which may be denoted as Adder FCOS . However , it is not easy to train the vanilla Adder FCOS in the same way as convolutional FCOS . Here we elaborate on some special designs for adder detectors . 3.1.1 Revisiting Batch Normalization in Adder Detector . Most modern object detection methods follow the paradigm of pre-training the backbone network on large scale image classification dataset ( e.g. , ImageNet ) and then fine-tuning the whole detector on the target detection dataset . In object detection , the batch size is usually much smaller than that of image classification tasks , due to relatively higher resolution of the input images . For example , generally there are only 2 or 4 images on one GPU for training an modern object detector . Therefore , the statistics of batch normalization ( BN ) are often frozen ( denoted as FrozenBN ) in the fine-tuning stage , which brings in a considerable improvement than the unfrozen counterpart ( simply denoted as BN here ) [ 6 ] . However , we empirically observe that FrozenBN in backbone leads to an unstable training of adder detectors . As shown in Fig . 3 , with FrozenBN in backbone , the training loss converges much slower than the unfrozen BN , resulting in a quite unsatisfied performance with mAP of zero . Therefore , updating the statistics of BN layers in backbone network is critical for training a detector built with adder filters . We attribute this phenomenon to the variance of features in AdderNets . As analyzed in [ 5 ] , the addition operations in AdderNets tend to have much larger variances for the features before batch normalization . Dong et al . [ 10 ] also analyze that the mean and variance of output feature are dominated by the variance of adder weights . Therefore , slightly tuning the adder filters would bring in drastic changing of the feature distribution , which makes the previous statistic of batch normalization incompatible with the input features . We visualize the statistics of batch normalization for a random layer in backbone during training in Fig . 2 . We can see that the weights for adder network ( Fig . 2c ) and convolutional network ( Fig . 2d ) exhibit quite different properties . The weights for convolutional network only have slight changes from the first epoch to the last epoch . How- ever , the weights for adder detector become much smaller as the training goes on . The changing of adder weights brings drastic variance for the output features , thus making the running means of BN for the last epoch quite different from that of the first epoch . If the statistics of BN are frozen during training , it would be quite challenging for training . To address this problem , it is necessary to unfreeze the statistic of batch normalization when finetuning the adder detectors from a pre-trained backbone . However , recomputing statistics for BN layers would be critical for the final performance , especially when the batch size is small , i.e. , 2 images per GPU . Therefore , it is necessary to use a larger batch size for training adder detectors .

Summary: The authors look at using AdderNets for object detection. AdderNets are neural networks which replace dot product between a weight filter and an activation window with the L1 distance between that weight and activation window instead. In hardware, AdderNets would replace multiplications with additions which are simpler and use less energy.  The authors use empirical experiments to discover a number of strategies that improve AdderNet AP, including: 1. Unfreezing the batchnorm statistics during downstream fine-tuning 2. Using direct gradients for the L1 distance (which are +`1/-1) instead of a continuous approximate gradient 3. Finetuning from a knowledge-distilled AdderNet backbone (as opposed to a regularly trained AdderNet) 4. Architectural changes specific to object detection, adding more skip connections  With these ideas, AdderNet versions of object detectors can approach 1-3% of their convolutional counterparts in AP while reducing multiplies by about 50% and energy consumption by `about 25%.