An Empirical Study of Adder Neural Networks for Object Detection
Year: 2022

Abstract:
Adder neural networks (AdderNets) have shown impressive performance on image classification with only addition operations, which are more energy efficient than traditional convolutional neural networks built with multiplications. Compared with classification, there is a strong demand on reducing the energy consumption of modern object detectors via AdderNets for real-world applications such as autonomous driving and face detection. In this paper, we present an empirical study of AdderNets for object detection. We first reveal that the batch normalization statistics in the pre-trained adder backbone should not be frozen, since the relatively large feature variance of AdderNets. Moreover, we insert more shortcut connections in the neck part and design a new feature fusion architecture for avoiding the sparse features of adder layers. We present extensive ablation studies to explore several design choices of adder detectors. Comparisons with state-of-the-arts are conducted on COCO and PASCAL VOC benchmarks. Specifically, the proposed Adder FCOS achieves a 37.8% AP on the COCO val set, demonstrating comparable performance to that of the convolutional counterpart with an about 1.4× energy reduction.

heading: 1 Introduction
text: Object detection is a foundational problem in computer vision and has attracted tremendous interests from both academic and industrial communities for decades [21]. It has a wide range of applications for various areas, e.g., video surveillance, autonomous driving and robotic vision. Deep neural networks have indeed dominated the research of object detection in recent years since the pioneering work of R-CNN [12]. The performance of object detectors has been considerably improved by various designs of architecture, loss function etc. However, most modern accurate object detectors require massive computation, making them quite challenging in resource-constraint applications, e.g., mobile phones and embedded devices.
Various approaches have been proposed to compress and accelerate convolutional neural networks (CNNs) for classification tasks, including channel pruning [24, 14, 26, 38], low-bit quantization [50, 25] and lightweight network design [33, 27]. These methods reduce the number of parameters or inference latency while maintaining the accuracy to the maximum extent. Such model compression methods have also been explored in a variety of down-stream tasks, such as semantic segmentation [7] and image super resolution [44] etc.
There have been a few methods aiming at fast and efficient object detectors. One family of solutions is using new architecture design [23, 31, 29, 41]. For example, YOLO series [30, 31] have achieved good trade-off between running speed and accuracy via a novel one-stage detection framework. Another family of solutions is to use common model compression methods for accelerating object
∗Corresponding author.
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
detectors, e.g., knowledge distillation [4, 42] and pruning [1]. Moreover, some recent works utilize neural architecture search (NAS) approach for searching better architectures for different components of object detectors [3, 13, 11]. Although these methods mentioned above show strong performance while improving the efficiency, they are mainly built with traditional convolutional neural networks, which contain massive inefficient multiplications.
Recently, Chen et al. [5] proposed the adder neural networks (AdderNets) to replace traditional convolutional filters with adder filters. Since addition is more energy efficient than multiplication [36, 47], AdderNets shed light on the design of efficient neural networks and have the potential of much fewer chip areas and less energy consumption. AdderNets have shown impressive performance in large scale image classification [45] via a kernel-based progressive distillation method, and also successfully been applied for other applications like image super resolution. Song et al. [34] proposed to utilize self shortcuts and learnable power activations to build super resolution networks via adder filters.
Existing variants of AdderNets mainly deal with either image classification [5, 45, 9] or superresolution tasks [34]. It is not clear yet how will the AdderNets perform for object detection, which often has sophisticated framework design and various objectives. This much more challenging computer vision task therefore brings in new challenges and opportunities for the research on AdderNets. So how to build accurate and efficient object detectors via AdderNets? The straightforward idea is to directly replace the original convolutional filters by adder filters. However, this naive adder detector cannot be easily trained as classification networks. At first, most modern detectors tend to pre-train the backbone on the ImageNet and then fine-tune the whole model on the target dataset, but this straightforward fine-tuning might worsen adder detectors, because of the sensitivity of adder filter. Moreover, the performance-proven neural architectures were all developed for convolution based detectors, and whether they are still applicable for the adder detectors is unclear.
In this paper, we propose a series of strategies to reform efficient object detectors with adder filters. In contrast with the frozen batch normalization widely exploited during fine-tuning of the detector, we empirically observe an opposite conclusion that adder detectors are better to unlock the statistics of batch normalization in pre-trained adder backbone for a performance improvement. Extensive ablation studies are conducted to explore the properties of batch normalization layers and the impact of batch size. In addition, a new feature fusion network with more residual connections and a better fusion module is explored to compensate for sparse adder features. Experimental results are reported on PASCAL VOC and COCO benchmarks, and the results are carefully analyzed and discussed. In particular, the proposed Adder FCOS achieves a 37.8% mAP on COCO val set, which is comparable with state-of-the-art object detectors, while saving much energy consumption as shown in Figure 1. In summary, we present an extensive empirical study for how to build object detectors via adder neural networks. We believe that the discussions and analysis in this paper will be beneficial for the research of efficient object detection and adder neural networks.

heading: 2 Related Work
text: Object Detection. Object detection has attracted tremendous interests for decades. Although there have been huge improvements for accurate object detection [21], the struggle for high-speed and energy-efficient detectors is largely unsolved. Several recent approaches exploit neural architecture search (NAS) to discover efficient and accurate object detectors [3, 13, 11]. On the meanwhile, there are some attempts to utilize model compression methods for improving the efficiency of detectors, e.g.,channel pruning [1], quantization [43] and knowledge distillation [4, 42]. Nevertheless, convolutional neural networks (CNNs) have in fact dominated the design of object detectors, which consist of massive multiplications and are quite energy inefficient.
Adder Neural Networks. Chen et al. [5] provided a new perspective for designing energy efficient neural networks. They proposed to utilize `1 distance to measure the similarity between the input features and filter weights. The devised AdderNets consist of only additions instead of massive multiplication, which are potentially energy efficient. Full precision gradients along with adaptive local learning rate are exploited to optimize this novel neural networks. AdderNets show impressive performance on large-scale image classification benchmark, i.e., ImageNet. To further improve the performance of AdderNets, Xu et al. [45] proposed a kernel based knowledge distillation method to learn from homogeneous CNNs. Using this method, AdderNets achieve even better accuracy than ResNet-50 on ImageNet. You et al. [47] further proposed to use bit-shift operations and additions to improve the energy efficiency. Song et al. [34] analyzed several key problems of AdderNets and successfully applied AdderNets for image super-resolution. Nevertheless, designing accurate object detectors with AdderNets is still less explored and is the main focus of this paper.

heading: 3 AdderNets for Object Detection
text: Traditional deep convolutional neural networks are mainly constructed by convolutional filters, which have massive manipulations and are energy inefficient. To this end, Chen et al. [5] proposed a new kind of neural architecture called AdderNet, which adopts `1-norm to compute the similarity between the input features X and the filter weights F , as shown in Eq. (1).
Y (m,n, t) = − d∑
i=0 d∑ j=0 cin∑ k=0 |X(m+ i, n+ j, k)− F (i, j, k, t)|. (1)
We aim to extend the success of AdderNet to the object detection task. We build our adder object detectors upon modern convolution-based detection frameworks, e.g., FCOS [39]. It is straightforward to replace the convolutional filters in the detector with adder filters. However, it is non-trivial to train an adder detector of decent performance. We first analyze several key strategies for applying adder filters for detectors. We then propose a novel multi-scale feature fusion architecture which is more suitable for adder detectors.

heading: 3.1 Making it Work: Towards a Strong Baseline
text: Intuitively if we replace the convolutional filters with adder filters in modern object detectors like FCOS [39], we could obtain a vanilla adder detector, which may be denoted as Adder FCOS. However, it is not easy to train the vanilla Adder FCOS in the same way as convolutional FCOS. Here we elaborate on some special designs for adder detectors.

heading: 3.1.1 Revisiting Batch Normalization in Adder Detector
text: Most modern object detection methods follow the paradigm of pre-training the backbone network on large scale image classification dataset (e.g., ImageNet) and then fine-tuning the whole detector on the target detection dataset. In object detection, the batch size is usually much smaller than that of image classification tasks, due to relatively higher resolution of the input images. For example, generally there are only 2 or 4 images on one GPU for training an modern object detector. Therefore, the statistics of batch normalization (BN) are often frozen (denoted as FrozenBN) in the fine-tuning stage, which brings in a considerable improvement than the unfrozen counterpart (simply denoted as BN here) [6]. However, we empirically observe that FrozenBN in backbone leads to an unstable training of adder detectors. As shown in Fig. 3, with FrozenBN in backbone, the training loss converges
much slower than the unfrozen BN, resulting in a quite unsatisfied performance with mAP of zero. Therefore, updating the statistics of BN layers in backbone network is critical for training a detector built with adder filters.
We attribute this phenomenon to the variance of features in AdderNets. As analyzed in [5], the addition operations in AdderNets tend to have much larger variances for the features before batch normalization. Dong et al. [10] also analyze that the mean and variance of output feature are dominated by the variance of adder weights. Therefore, slightly tuning the adder filters would bring in drastic changing of the feature distribution, which makes the previous statistic of batch normalization incompatible with the input features. We visualize the statistics of batch normalization for a random layer in backbone during training in Fig. 2. We can see that the weights for adder network (Fig. 2c) and convolutional network (Fig. 2d) exhibit quite different properties. The weights for convolutional network only have slight changes from the first epoch to the last epoch. How-
ever, the weights for adder detector become much smaller as the training goes on. The changing of adder weights brings drastic variance for the output features, thus making the running means of BN for the last epoch quite different from that of the first epoch. If the statistics of BN are frozen during training, it would be quite challenging for training.
To address this problem, it is necessary to unfreeze the statistic of batch normalization when finetuning the adder detectors from a pre-trained backbone. However, recomputing statistics for BN layers would be critical for the final performance, especially when the batch size is small, i.e., 2 images per GPU. Therefore, it is necessary to use a larger batch size for training adder detectors.

heading: 3.1.2 Better Pre-trained Backbone and Gradients
text: The performance of vanilla AdderNet [5] on ImageNet is still poorer than CNNs. For better performance, we use the pre-trained model using the knowledge distillation proposed in [45]. The Top-1 accuracy of AdderNet-50 on ImageNet is 76.8%, which is comparable with the performance of its convolutional counterpart ResNet-50. More discussions for the pre-trained models will be included in the ablation studies.
The original AdderNets [5] require a special design of gradient descent rule for optimization. More specifically, instead of using the actual sign gradients, AdderNets exploit the full precision gradients of `2-norm to update the filters. To avoid the magnitude accumulation brought by the gradient chain rule, AdderNets clip the gradients of input features X to [−1, 1] using a HardTanh function. Though this practice works well in image classification tasks, we argue that this gradient approximation may introduce challenges for optimization for object detection which often exhibits more complicated architectures. Since the gradients of X only play the role of accumulate gradients in chain rule, it is better to use the sign gradients for X . Therefore, we exploit the gradients of `2-norm for adder weights and sign gradients for the input features.

heading: 3.2 Better Feature Fusion for Adder Detector
text: Using all strategies discussed in the above sections, we obtain a baseline Adder FCOS with the mAP of 34.8%, which is quite close to the convolutional counterpart. We then move one step forward to explore a better architecture for adder detectors.
We first visualize features from the pretrained backbone, as shown in Fig. 4. The feature maps from the last block of AdderNet-50 is much sparser than ResNet-50. More specifically, over 92% of the features for the last output from AdderNet-50 are zeros, while the percentage for ResNet-50 is only 63%. These sparse features may be enough for image classification task. However, it poses great challenges for object detection which needs dense predictions for class classification and bounding box regression.
We attribute the problem of sparse features to the calculation of adder filters. As shown in Eq. (1), the output features of adder operation are always negative. Although the normalization procedure in BN makes these features have the mean of zero and the variance of one, the scaling and shift parameters in BN are learned to restore the representation power of the original features and tends to move features towards negative. The following ReLU activation would eliminate the negative features, making features in deeper layers to be more sparse.
Multi-scale feature fusion module is widely adopted for accurate object detection [19, 16, 22, 11]. As shown in Fig. 5 (a), feature Pyramid Network (FPN) [19] utilizes a top-down architecture to aggregate features from different levels and enhance the high-level semantic features for all scales. PAFPN [22] added extra bottom-up path for better feature aggregation as shown in Fig. 5 (b). The feature fusion module somehow alleviates the feature sparsity problem of adder detectors. However, since the pre-trained model of AdderNet has sparse features in top layers, the top-down path may not bring meaningful information for feature fusion.
Therefore, we hereby propose a novel feature fusion module to alleviate this problem. Unlike most previous multi-scale feature fusion methods [19, 22, 37] that first adopt top-down path for feature aggregation, we propose to exploit a reverse pattern. Specifically, the proposed R-PAFPN first exploits bottom-up path to propagate features from bottom layers to top layers, and then utilizes top-down path for fusing semantically strong features. This simple yet effective design is more suitable for adder detectors. Moreover, as discussed in [34], identity mapping is challenging for AdderNets. Therefore, we also add extra skip connections in our proposed feature fusion module. Specifically, we add residual connections for each 3x3 adder filters for better feature propagation, as shown in Fig. 5.

heading: 4 Experiments
text: In this section we first conduct extensive ablation experiments to analyze the effectiveness of different components of the proposed method. After that, we compare our proposed method with state-of-theart object detectors.

heading: 4.1 Experimental Settings
text: We conduct experiments on the bounding box detection track of MS COCO 2017 and PASCAL VOC benchmarks, which have 80 and 20 object classes, respectively. On all experiments, the AdderNet backbone keeps the first layer as convolution and has all the rest layers built with adder filters, following the practice in [5, 45].
COCO. Following the common practices, for COCO benchmark we use the COCO train2017 split that contains 118k images for training, val2017 split for validation (5k) and test-dev split (20k) for testing. We report the average precision (AP) w.r.t. different IoU thresholds and different object scales, i.e.,mAP, AP50, AP75, APS , APM and APL. All models are trained with stochastic gradient descent (SGD) over 8 GPUs. Unless otherwise specified, all models are trained for 12 epochs (also known as 1× schedule) with cosine learning rate decay strategy. Weight decay and momentum are set to 0.0001 and 0.9, respectively.
PASCAL VOC. For PASCAL VOC benchmarks, we train our models on the VOC 2007 and 2012 trainval sets, which contain about 16, 551 images, and evaluate on the VOC 2007 test set (4952
images). We use the VOC style mAP (i.e.,mAP at IoU=0.5) as the evaluation metric. The input images are resized to have shorter side being 600 while the longer side not to exceed 1000. Other hyper-parameters are similar to COCO benchmark. Our implementation is based on the popular object detection framework MMDetection [6].

heading: 4.2 Ablation Studies
text: Steps towards a strong baseline. We first analyze the strategies introduced in Section 3.1 on FCOS with adder backbone and neck. As shown in Table 1, if we train an adder FCOS detector with FrozenBN, the loss could not converge properly and the detector gets zero mAP. Updating the statistics of running mean and variance for BN (i.e., Unfrozen BN) is critical for training the adder detector and achieve reasonable mAP of 32.4%. We further exploit `1 gradients for input features instead of clipped `2 gradients as in [5], which harvest about 0.2 mAP improvement.
To further improve the performance of the adder detector, we instead use a better pre-trained backbone, which is trained on ImageNet via kernel based knowledge distillation [45]. This strategy boosts the mAP to 33.2%. We also introduce cosine learning rate and hyper-parameter tuning for learning rate, which obtain 0.6% mAP improvement. Now we have got a strong baseline for adder detector, which achieves 34.8% mAP on COCO val2017 set.
The AdderNet-50 trained with knowledge distillation [45] has 76.8% top-1 accuracy on ImageNet, which is slightly higher than that of ResNet-50. To explore to what extent the pre-trained backbones affect the final detection performance, we evaluate the backbones with and without knowledge distillation, as shown in Table 2. Using a vanilla AdderNet-50 with 1.9% top-1 accuracy drop on ImageNet suffers only 0.4% mAP drop compared with the KD counterpart. The AdderNet-18 trained with KD and the vanilla AdderNet-101 perform worse than their convolutional counterparts on ImageNet but still obtain considerably good performance for object detection.
Impacts of batch size. As discussed in Section 3.1.1, it is necessary to recompute the statistics of batch normalization layers when training the adder detectors. We first explore the performance of convolutional detectors with different batch sizes. As shown in Figure 6, when the batch size is relatively large enough, i.e., batch size of 8, the performance of using FrozenBN and normal BN are similar. Reducing the batch size to 4 or 2, the accuracies of using BN
(i.e., recomputing statistics) decrease drastically while the counterpart with FrozenBN suffers slight performance degradation. Similar results are observed for adder detectors with normal BN, which demonstrates the intuitive fact that training detectors when recomputing statistics for BN with small batch size is challenging. Moreover, adder detectors suffer severer accuracy drop when reducing the batch size, e.g., 5.4 mAP drop when decreasing batch size from 8 to 2 for adder detector while only 3.4 mAP drop for convolutional one. We perform more analysis on several modern detectors, as shown in Table 3. We replace the backbone of detectors with adder networks and explore the performance with different batch sizes. For most detectors, larger batch size consistently improves the performance, which demonstrates that large batch size is critical for training adder detectors.
Impacts of different neck structures. In Section 3.2, we propose a novel multi-scale feature fusion architecture (R-PAFPN) for improving the performance of adder detectors. Here we elaborate the effectiveness of the proposed design choices. Original FCOS [39] adopts feature pyramid network (FPN) for feature aggregation. Its adder counterpart achieves mAP of 34.8%, as shown in Table 4. We simply replace the neck architecture to PAFPN [22], which adds an extra bottom-up feature fusion path to FPN. This modification only brings minor improvement, i.e.,0.1% mAP, which demonstrates that simply exploiting more fusion paths is not enough for adder detectors and special design is urgent.
We further add extra shortcut connections to PAFPN (denoted as PAFPN w/ shortcut), motivated by the discussions in AdderSR [34]. Experiments show that adding skip connections is also beneficial
for detection task, which bring 1.3% mAP improvement. As discussed in Section 3.2, the pre-trained backbone for adder detector exhibits sparser feature maps for deeper layers. Therefore, we propose to utilized bottom-up feature aggregation path to fuse features of different scales from the backbone network. As shown in Table 4, the proposed R-PAFPN neck structure harvests additional 0.9% mAP gains, achieving the mAP of 37.0% on COCO val2017 set. It justifies our motivation that bottom-up path should be applied first since the low-level features could compensate for the sparse high-level features.

heading: 4.3 Experiments on COCO
text: We evaluate our proposed method on COCO benchmark for several state-of-the-art object detectors, including FCOS [39], FoveaBox [15], RetinaNet [20] and Sparse R-CNN [35]. We replace various components for these detection frameworks, e.g., Backbone (B), Neck (N) and Head (H), as shown in Table 5. For example, FCOS with adder backbone and neck is denoted as Adder FCOS (B+H). Similar to prior method [34], we calculate the energy costs of different detectors. As discussed in prior literature [8, 47], one operation of floating-point addition and multiplication have energy costs of 0.9 pJ and 3.7 pJ , respectively.
For standard 1× schedule (i.e., 12 epochs), our adder detectors with AdderNet-50 backbone and convolutional neck, achieve comparable performance on COCO val2017 set with their convolutional baselines. For example, Adder FCOS (B) achieves mAP of 37.2% , which is only 1.2% mAP lower than FCOS [39]. Since the backbone is built with adder filters which get rid of massive multiplications, Adder FCOS (B) has considerably fewer number of multiplications than FCOS (129.9 vs. 214.7). Sparse R-CNN [35] with adder backbone only suffers 0.9% mAP drop while reducing the potential energy costs from 719.5 to 482. We also build a detector with adder backbone and neck, i.e., Adder FCOS (B+N), which achieves 37.0% mAP and have further fewer multiplications. Compared with RetinaNet [20], our adder FCOS achieves the same detection accuracy but have much higher potential for energy efficiency.
We also conduct experiments on longer schedule (i.e., 2×). Adder FCOS (B+N) achieves 37.8% mAP and outperforms RetinaNet [20] by 0.4% mAP. It obtains quite competitive performance with FoveaBox [15], Faster R-CNN [32] and RepPoints [46] but have much fewer multiplications. For ResNet-101 backbone, the adder counterpart suffers from 2.4% mAP drop while having 1.6× energy reduction.
Figure 7 shows some qualitative results of our proposed adder detectors and state-of-the-arts detectors, including RetinaNet [20] and FCOS [39]. Adder FCOS works well for a variety of challenging scenarios and have similar predictions with other detectors.

heading: 4.4 Experiments on PASCAL VOC
text: We also evaluate our proposed method on PASCAL VOC dataset. Our Adder FCOS with adder backbone and neck structure achieves mAP of 76.5, which is comparable with FoveaBox [15] and RetinaNet [20]. The performance is a bit poorer than Faster R-CNN and FCOS. However, considering the energy cost reduction, the proposed adder detector is a good trade-off for object detection accuracy and energy efficiency.

heading: 5 Conclusion and Discussion
text: In this paper, we present an empirical study for accurate object detectors via adder neural networks. We first reveal that unfreezing statistics of batch normalization in backbone is crucial for adder detectors. We empirically analyze the properties of batch normalization and the impact of batch size. We also move an extra step forward to exploit a better architecture for adder detector. More specifically, we propose a novel reverse multi-scale feature fusion module called R-PAFPN, which compensates for the sparse high-level features with feature aggregation. Extensive experiments are conducted on COCO and PASCAL VOC benchmarks. In details, Adder FCOS achieved 37.8% AP on COCO val set, demonstrating comparable performance with convolutional counterpart but having much more potential energy reduction.
There are still some unsolved limitations for adder detector. For example, it still suffers from a bit accuracy degradation when compared with its convolutional counterpart, i.e., FCOS [39]. What’s more, the prediction heads on some adder detectors are still stacked with convolutions, and we empirically find that it would bring considerably large performance reduction if we replace convolutional filters on heads with adder filters for FCOS. Nevertheless, considering the energy cost saving brought by AdderNets, it is still a promising solution for efficient object detectors. In further work, it would be interesting to design a new prediction head architecture which is suitable for accurate adder detectors. We hope this study will be helpful for the research of adder neural networks and energy-efficient object detection.
Acknowledgments and Disclosure of Funding
The authors sincerely thank anonymous reviewers and ACs for their helpful comments. Chang Xu was supported in part by the Australian Research Council under Projects DE180101438 and DP210101859.

Review summaries:
This paper is an empirical study of using adder networks for the task of object detection. These networks (mostly) use additions over multiplications, and are a good choice for improving computational performance for this task, specially for edge and embedded devices. I belive that the authors in this paper only consider adder networks only for the backbone and neck architectures of an object detector, leaving out the prediction heads which still use full convolution.  The contributions are: some tricks and strategies to train AdderNet's for object detection, a comprehensive evaluation on COCO and PASCAL VOC, a residual version of FPN which has slightly better performance, knowledge distillation for an improved backbone network for the object detector, and an analysis of freezing batch normalization and the effect of the batch size on the adder object detection networks. 

The paper investigates the use of AdderNets (networks that do not use multiply operations) for object detection. Several modifications to the original AdderNets architecture are proposed including "unfreezing" batch norm statistics during inference as well as residual connections. Knowledge distillation as well as "other tricks" (L206) are used to further improve performance. Results are reported for some modified (replace traditional convs with adders) detection architectures are presented. The results show a moderate drop in performance, while supposedly using less power (estimated in mJ). 

This paper empirically explores the impact of adder-based network architecture for developing object detectors. It studies the impact of batch normalization and batch size on the performance of adder networks and proposes unfreezing batch normalization for adder networks. It also explored the effect of different neck architectures by investigating the limitations of adder networks and how to improve them. 

The authors look at using AdderNets for object detection. AdderNets are neural networks which replace dot product between a weight filter and an activation window with the L1 distance between that weight and activation window instead. In hardware, AdderNets would replace multiplications with additions which are simpler and use less energy.  The authors use empirical experiments to discover a number of strategies that improve AdderNet AP, including: 1. Unfreezing the batchnorm statistics during downstream fine-tuning 2. Using direct gradients for the L1 distance (which are +`1/-1) instead of a continuous approximate gradient 3. Finetuning from a knowledge-distilled AdderNet backbone (as opposed to a regularly trained AdderNet) 4. Architectural changes specific to object detection, adding more skip connections  With these ideas, AdderNet versions of object detectors can approach 1-3% of their convolutional counterparts in AP while reducing multiplies by about 50% and energy consumption by `about 25%.