Title: An Empirical Study of Adder Neural Networks for Object Detection
Authors: Xinghao Chen
Chang Xu
Minjing Dong
Chunjing Xu
Yunhe Wang
Venue: NeurIPS 2021
Keywords: Object Detection
Adder Neural Networks

Abstract: Adder neural networks (AdderNets) have shown impressive performance on image classification with only addition operations, which are more energy efficient than traditional convolutional neural networks built with multiplications. Compared with classification, there is a strong demand on reducing the energy consumption of modern object detectors via AdderNets for real-world applications such as autonomous driving and face detection. In this paper, we present an empirical study of AdderNets for object detection. We first reveal that the batch normalization statistics in the pre-trained adder backbone should not be frozen, since the relatively large feature variance of AdderNets. Moreover, we insert more shortcut connections in the neck part and design a new feature fusion architecture for avoiding the sparse features of adder layers. We present extensive ablation studies to explore several design choices of adder detectors. Comparisons with state-of-the-arts are conducted on COCO and PASCAL VOC benchmarks. Specifically, the proposed Adder FCOS achieves a 37.8% AP on the COCO val set, demonstrating comparable performance to that of the convolutional counterpart with an about $1.4\times$ energy reduction.

TL;DR: An emperical study of how to use adder neural networks to build accurate object detectors.

Review discussion:
The proposed work applies the recently published AdderNets to the detection problem. The modifications proposed are highly ad-hoc, and the motivations behind them are questionable.
Specifically, I do not find the results shown in Figure 2 convincing for unfreezing the batchNorm params during inference. I would appreciate at least an attempt at some analysis as to why training becomes unstable without unfreezing BatchNorm. Why are AdderNets more sensitive than traditional convnets? Perhaps start by noting that: exp(log(a) + log(b)) = a*b.
Also proposing to unfreeze these params is highly limiting, for example consider a mobile device setting: can inference even be done on one image at a time (i.e. single sample batch stats are unreliable)?
Finally, the baseline models used by the authors are far from modern state of the art (https://paperswithcode.com/sota/object-detection-on-coco) which severely limits the impact of these results.
No
We thank the reviewer for the constructive comments.
Before getting into the detailed responses, we want to clarify that the proposed method unfreezes the BatchNorm statistic only during TRAINING, rather than the inference that was summarized by the reviewer (see the caption of Figure 2 and line 127).
This paper extends the success of AdderNet to the detection task by developing adder detectors.
However, it is very hard to directly replace convolutional filters in an object detectors by adder filters while maintaining the original performance. To this end, in this paper we present an extensive empirical study for developing accurate adder detectors.
In particular, a bag of strategies are carefully designed including the unfreezing BatchNorm (Section 2.1) and a new feature fusion network (Section 2.2), etc. Extensive experiments on the COCO benchmark dataset and various detectors demonstrate the effectiveness of the proposed method.
**Results in Figure 2**
In Figure 2, we visualize that the statistics of BatchNorm change drastically for the beginning and the end of training, which indicates that the adder detector prefers quite different BatchNorm statistics from ImageNet pre-trained classification backbone. If we still freeze the BatchNorm statistics when fine-tuning the whole detector, there will be a large gap between the desired running mean/variance with the pre-trained one. This incompatibility brings severe challenge for training adder detectors. This phenomenon may be mainly due to the drastic changing of adder weights, which affects the output features thus make the BatchNorm changing rapidly, as shown in Figure 2(c).
**Some analysis as to why training becomes unstable without unfreezing BatchNorm**
Most modern object detection methods follow the paradigm of pre-training the backbone network on large scale image classification dataset (e.g. ImageNet) and then fine-tuning the whole detector on the target detection dataset. For convolutional detectors, it does not matter too much whether to freeze the BatchNorm statistic or not, when the batch size is large enough as shown in Figure 6.
However, things become quite different for adder detectors, in which the unfrozen BatchNorm is essential for convergence. We include some analysis below.
Denote the output of the $l^{th}$ layer as $ y'_l = -\sum^n |x_l-W_l|$, where $W_l$ is the weight and $x_l$ is the input feature. $W_l$ follows Laplace distribution $\mathcal{L}(\mu,b)$ with the mean as $\mu$ and the variance as $2b^2$. We can deduce that the mean of $y_l$ can be computed as $-n(b+\epsilon)$ and the variance as $n(b+\epsilon)^2$, where $\epsilon$ is a value that has similar order of magnitude as the variance of $x_l$. Since $x_l$ is the output of last layer and goes through the BatchNorm layer, $\epsilon$ is relatively smaller.
Therefore, the mean and variance of $y_l$ are dominated by $b$, i.e., the variance of $W_l$. As shown in Figure 2, the variance of $W_l$ is quite large, making the mean and variance of $y_l$ also large.
Therefore, the statistics of BatchNorm have to be recalculated when fine-tuning the adder detectors from a pre-trained backbone. We will include more related analysis in the final version.
**Can inference even be done on one image at a time (i.e. single sample batch stats are unreliable)**
Sorry for the misleading, we would like to clarify that the proposed method only unfreezes the BatchNorm during the training of detector network, which is essential for the adder detectors to converge. During inference, the BatchNorm is set as same as that in modern standard detectors. More specifically, after training all parameters for BN layers will be fixed for the adder detectors and will be used for inference.
Therefore, we do not need to calculate BN statistic during inference, thus adder detectors are also suitable for single image batch.
**State-of-the-art**
We are fully aware that recent state-of-the-art detection performance on coco dataset has reached quite high mAP.
However, more training strategies, more complicated backbones or architecture design have to be adopted to achieve such detection accuracy.
For example, current top 9 SoTA detectors listed in [https://paperswithcode.com/sota/object-detection-on-coco](https://paperswithcode.com/sota/object-detection-on-coco) utilized vision transformer (i.e. Swin Transformer) as backbones. We mainly focus on developing adder networks for convolutional calculation, and how to apply adder filters to vision transformers/attention module is still an open yet interesting topic, which is out of scope of this manuscript.
Besides, the currently best CNN-based detector is Cascaded EfficientNet-B7 NAS-FPN. It utilizes more complex architecture such as cascaded framework, EfficientNet-B7 and NAS-FPN.
Moreover, it adopts several training techniques like Copy-Paste augmentation and self-training. As also pointed out by Reviewer 2TRF, obtaining this kind of performance requires a lot of computing resources and engineering effort.
To make our evaluation more clear and demonstrate the proposed method on a neater baseline, we mainly evaluated the proposed method upon recent classical and yet competitive detectors, including RetinaNet (ICCV 2017), FCOS (ICCV 2019, TPAMI 2020), Sparse R-CNN (CVPR 2021) etc. For neat comparisons, we adopt widely used ResNet backbone with simple training schedules and demonstrate the effectiveness of the adder detectors on various detection frameworks.
However, we believe the conclusion drawn from this manuscript can also be compatible with more complicated detector architecture and training tricks.
Therefore, we leave it as future work to achieve higher mAP with modern detection methods for adder detectors.
Thank you for the clarification regarding BatchNorm stats being updated during training only. I maintain my score as I believe this paper is of limited interest to the wider NeurIPS community and may be more well suited to a computer vision conference (ICCV/CVPR) as it's main contributions are various engineering tricks for highly specialized use cases (e.g. mobile devices).
Dear Reviewer,
Thanks a lot for your review comments and we are happy to see that the confusion of BatchNorm has been clarified in the previous response.
In this paper we perform a thorough empirical study of adder neural networks for object detection. To the best of our knowledge, this is the first attempt to explore how to build energy-efficient object detectors with adder networks. We present extensive analysis and experiments for several important issues for adder detectors.
Accordingly, we provide a bags of good practices and novel strategies for building accurate object detectors via adder neural networks.
We believe that such paper is of broad interest to the NeurIPS community.
In particular, there have been several empirical studies for different topic in previous venue of NeurIPS [1,2,3,4]. AdderNet is a novel design of energy-efficient network and has also inspired several follow-up research works that were published in NeurIPS [5,6].
Besides, object detection is a foundational computer vision problem and has consistently received research interest from NeurIPS community [7,8,9]. We hopefully believe that this paper, as the very first attempt to empirically and thoroughly explore the problem of building accurate and energy-efficient object detectors with adder networks, will be beneficial for the community of NeurIPS.
[1] How Can I Explain This to You? An Empirical Study of Deep Neural Network Explanation Methods, NeurIPS 2020
[2] Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the Neural Tangent Kernel, NeurIPS 2020
[3] Finite Versus Infinite Neural Networks: an Empirical Study, NeurIPS 2020
[4] Failing Loudly: An Empirical Study of Methods for Detecting Dataset Shift, NeurIPS 2019
[5] ShiftAddNet: A Hardware-Inspired Deep Network, NeurIPS 2020
[6] Kernel Based Progressive Distillation for Adder Neural Networks, NeurIPS 2020
[7] Detection as Regression: Certified Object Detection with Median Smoothing, NeurIPS 2020
[8] RepPoints v2: Verification Meets Regression for Object Detection, NeurIPS 2020
[9] RelationNet++: Bridging Visual Representations for Object Detection via Transformer Decoder, NeurIPS 2020
Strengths:
1. AdderNets for object detection is an interesting and worth studying direction. This work studies this problem for the first time.
2. The technical part is well written.
Weaknesses:
1. This paper claims that CNN contains massive inefficient multiplications, which leads to more energy consumption than AdderNets.
But I cannot see inference speed comparison between AdderNet-based and CNN-based detectors. Inference speed is an important metric in real-world applications.
2. BN statistics change drastically when the weights pre-trained on ImageNet are used in COCO, as shown in Figure 2.
Therefore, the AdderNet-based detector seems to be less robust to the domain shift.
3. The AdderNet-based detector uses some tricks (see Table 1) to improve the performance. These tricks can also be used to improve CNN-based models.
But the reported CNN-based method does not include these improvements.
4.  The author provides some ways to alleviate these problems, but I think there will be more essential solutions, such as avoiding these problems when designing the adder filter.
No
We thank the reviewer for the constructive comments.
**Inference speed comparison between AdderNet-based and CNN-based detectors**
Thanks for the nice concern. In general hardware like GPU and off-the-shell deep learning frameworks, adder networks do not exhibit faster inference speed due to the fact that the matrix multiplication has been highly optimized in CUDA and GPU, while the adder operations are not.
However, some recent work have demonstrated that AdderNet can achieve 16% speedup in FPGA [R1] (see Section 4, P8) for ResNet-18. The proposed AdderNet-based detectors also utilize ResNet as backbone, thus they should also achieve some speedup. We also would like to emphasize that the main advantage of adder networks lies on the energy cost and chip area reduction on custom-tailored hardware, while AdderNets would not present huge inference speedup.
[R1] AdderNet and its Minimalist Hardware Design for Energy-Efficient Artificial Intelligence, arXiv: 2101.10015, 2021
**AdderNet-based detector seems to be less robust to the domain shift**
Thanks for this constructive comments. It is an interesting topic to explore the robustness to the domain shift for AdderNet-based detector. We utilize the FCOS models trained on COCO to evaluate on the Cityscapes dataset for the common object categories without fine-tuning. More specifically, we adopt the coco-trained models to predict the detection results for Cityscapes val set, and map the original ground-truth categories of Cityscapes to COCO (80 classes) to calculate the mAP. Adder FCOS suffers from 2.2 mAP drop on Cityscapes, which is similar with the performance drop on COCO, demonstrating that adder detectors have similar robustness performance to the domain shift with CNN detectors. We will include more results and discussions in the final version.
|    Model               |    Coco mAP       |    Cityscapes mAP    |
|------------------------|-------------------|----------------------|
|    FCOS                |    38.4           |    29.4              |
|    Adder FCOS (B+N)    |    36.5 (-1.9)    |    27.2 (-2.2)       |
**These tricks can also be used to improve CNN-based models**
The training tricks in Table 1 mainly include larger learning rate with cosine decay. These tricks bring 0.2-0.6 mAP gain for various CNN-based detectors. We will include these results in the final version.
In addition, the unfrozen BatchNorm is very useful to train adder detectors while useless to train cnn detectors as shown in Figure 3 and Figure 6.
|    Model  | step lr | cos lr + larger lr |
|-----------|---------|--------------------|
| RetinaNet | 36.4    | 36.6               |
| FCOS      | 38.4    | 39.0               |
**There will be more essential solutions, such as avoiding these problems when designing the adder filter**
Thanks for the nice concern. In this paper we focus on extending AdderNet that is trained on large scale image classification dataset to downstream object detection task.
Therefore, we simply adopt the adder filter from the original AdderNet. Designing a better adder filter is a bit out of scope of this paper. Through extensive analysis and experiments, we present that there are several issues of the original AdderNet for object detection.
Therefore, we propose a bag of strategies to obtain better performance for adder detectors.
Nevertheless, we agree with the reviewer that it is an interesting topic to design a new adder filter that has more powerful performance for different tasks and still enjoys the multiplication-free properties. We leave this topic as future work and will include related discussions in the manuscript.
Dear Reviewer,
Can you please update your review or respond to the author's rebuttal? It will be good to clarify if the authors have addressed your concerns and what concerns still hold.
AC
The author's feedback partially solves my concerns.
But I am still confused why the adder detector has similar robustness performance to the domain shift with the CNN detector. As shown in Figure 2, CNN's BN statistics are more stable than AdderNet.
In addition, although this is the first attempt to explore how to build an object detector with an adder network, the neck and detection head are still based on convolution.
So I think this work is below my expectations.
Dear Reviewer,
Thanks for your review comments and we are happy to see that some concerns have been clarified in the previous response.
We agree that the domain shift is a quite interesting topic for adder detectors. As shown in the above response, adder detectors show similar robustness performance to the domain shift with CNN detectors. It may due to that the unstable BN statistics only exist in training phase. Once the detector is trained, the behavior of BN layer is the same with CNN detectors.
Therefore, if we use the detector for other domain, the performance of adder detectors is likely similar to that of CNN detectors. Investigating the domain shift performance of object detectors is an interesting yet complicated problem and we leave more analysis as future work.
What is more, we would like to clarify that the proposed adder detectors are not limited to adder backbone. As shown in Table 5, the performance of various detectors with adder backbone and neck (B+N) is quite competitive.
Besides, we show the performance of RetinaNet with adder backbone, neck and prediction head in the last row of Table 5, which achieves over 2x energy reduction with around 3.4 mAP drop.
Positives
- The paper is well motivated, as object detection computational performance is an important target for improvement, and is very important for edge devices. AdderNets are a good candidate for performance improvement as they are a meta-architecture and reducing the number of multiplications makes a lot of sense for good computational performance.
- The paper is very well written, with good use of space and words, including a lot of results and ablation studies that support the paper's conclusions and overall scientific decision making.
- The evaluation has a good selection of methods from the state of the art and detectors that are transformed into its AdderNet version. I have no complains about the overall evaluation, it looks correct. All decisions are being correctly motivated and justified with a scientific experiment.
- While there is no improvement to the state of the art in terms of mAP, there is improvement in terms of number of additions and multiplications, and overall amount of energy required for a forward pass, and this means a small sacrifice in mAP, both in COCO and PASCAL VOC (around 3 mAP points).
Negatives
- There seems not to be any comparison to other techniques that also target a reduction in computational cost, some of these techniques are mentioned in the introduction, like pruning, weight quantization, and lightweight network architectures. This comparison would make the paper much more strong, in order to connect with the original motivation of computational performance in object detection.
- The whole evaluation metric for computational performance is the amount of energy required for a forward pass, I believe that the computation time on a standard platform would also be useful. The number of FLOPs can also be inferred from number of additions and multiplications, and this metric can also be plotted directly with raw mAP. I have more suggestions below in the minor comments on how to provide additional views of your results.
- Overall performance of the best model (mAP 37.8% in COCO) is quite far from the greater state of the art, where in the COCO minival the best performance  I could find according to [PapersWithCode](https://paperswithcode.com/sota/object-detection-on-coco-minival?tag_filter=12%2C13) is around 57.0% mAP. This is only a minor issue since obtaining this kind of performance requires a lot of hacks that are not being applied here.
- In the conclusion it is mentioned that full adder networks (including prediction heads) would also provide additional computational performance improvement, but why were these results left out of the paper?
Minor Comments
- In Figure 1, I think it would be better to use different marks (diamond, triangle, sqiuare) for the original object detection method and its Adder variant, while using different colors for different object detection methods. This would make Figure 1 easier to read and understand.
- Table 4, please mention in the caption on which dataset these results were obtained.
- It would be very interesting to plot the results of Table 5, I am aware that some of these results are included in Figure 1, but for example, the trade-off between performance and computation could be plotted as mAP vs Energy, or number of additions vs number of multiplications, with different colors for a scatterplot that represents mAP performance, etc. This could show the trade-off between # of additions, # of multiplications, and performance in terms of mAP, for a large selection of detectors. This could be added to the supplementary material.
- One thing I notice in Figure 7 is that overall (not always), seems that FCOS and Adder FCOS are less confident (by looking at the bounding box probabilities) than RetinaNet, and Adder FCOS seems to be a bit less confident than FCOS. This is something to consider in the future as detection confidences are also quite important.
- The paper could benefit from supplementary material, for example, expanding the results in Figure 7 to all detectors and a larger selection of images both for PASCAL VOC and COCO.
Overall I believe this is a worthy contribution to the state of the art and should be accepted.
No
Thanks for your strong support.
**Comparison to other techniques that also target a reduction in computational cost**
Thanks for the good suggestion. We compare with the pruning results in Table 2 of Joint-DetNAS (Yao et al., CVPR 2021). The results of Joint-DetNAS are directly borrowed from its published version.
When pruning about 30% channels of the ResNet-50 backbone (i.e. 221.02 #MUL, 221.02 #ADD and 1017 mJ energy with only 8% energy reduction), the detector has an about 0.7 mAP drop.
In contrast, our detector using adder backbone has over 25% energy reduction while only suffering an about 0.8 mAP drop, demonstrating better energy-accuracy trade-off. We will include more results and discussions in the final version.
Furthermore, AdderNet can apply for pruned architecture and channel pruning methods can also utilized for pruning AdderNet as discussed in AdderSR (CVPR 2021), which is an interesting topic for future direction.
**The computation time on a standard platform, The number of FLOPs**
1)	In general hardware like GPU and off-the-shell deep learning frameworks, adder networks do not exhibit faster inference speed due to the fact that the matrix multiplication has been highly optimized in CUDA and GPU, while the adder operations are not.
However, some recent work have demonstrated that AdderNet can achieve 16% enhancement for speed in FPGA (AdderNet and its Minimalist Hardware Design for Energy-Efficient Artificial Intelligence, arXiv: 2101.10015, 2021).
2)	Previous methods usually treat FLOPs as the number of one multiplication and addition operations. Since mul-add is a standard operation in convolution, FLOPs is widely used in CNN to indicate the computational costs. In AdderNets, only addition is adopted so the definition of FLOPs for AdderNets is a bit unclear.
Therefore, we report the energy cost instead, which is highly related to the number of multiplication and addition.
**State of the art performance**
Thanks for the comments. We will try to conduct more experiments for detectors with state-of-the-art performance.
**Full adder networks**
We have provided full adder network (including adder backbone, neck and prediction head) in Table 5 for RetinaNet, which achieves 3.4% mAP drop with about 50% energy cost. As we also discussed in the conclusion part, replacing prediction head with adder filters may lead to considerably large performance drop for some detectors, e.g., FCOS. We believe this is an interesting topic and leave it as future work.
**Minor comments**
-	Thanks for the great suggestions. We will update Figure 1 accordingly.
-	We have changed the caption of Table 4 to “Ablation studies for neck structures on COCO *val2017*” to indicate the dataset.
-	Thanks for the good suggestions. We will plot the results of Table 5 as suggested and include it in the supplementary material.
-	The results shown in Figure 7 did demonstrate that in some cases Adder FCOS is a bit less confident than FCOS and we think this is mainly due to slight lower mAP for Adder FCOS.
Besides, these results show that FCOS/Adder FCOS are less confident than RetianNet. By analyzing more visualization results, we find that it is not always true and we will include more visualization in the final version.
-	Thanks for the good suggestions and we will include more visualization results for PASCAL VOC and COCO dataset in the supplementary material.
Thank you for the additional details, I will keep my score. One additional comment from my side, other reviewers have mentioned that this paper could also fit into a computer vision conference, and I agree. That should not be a reason for rejection, but it is something to consider for the next version.

Meta-review:
role: PC
data: comment: The crux of the paper is to enable object detection using Adder Networks -- architectures optimized for energy efficiency. The paper proposes a couple of empirically validated heuristics that certainly achieve the objective. Consequently, reviewer 2TRF argues that the paper should be accepted. However, reviewers 3Btm and cnqS argue that the paper should be rejected.

- I acknowledge Author's concerns about cnqS's opinion that this paper is not a good fit for NeurIPS. I don't think cnqS's concerns in this context are valid. 

- The main concern of 3Btm is that the proposed changes are trivial. I don't consider the proposal to be trivial, since it does lead to an increase in performance. After all, even the AlexNet paper talked about specific ways of tuning / architectural choices to enable CNN training. 

My main concern really is that there are many ways of increasing energy efficiency -- reduced precision, distillation, pruning, etc. 2TRF raised this point in the review and authors compared the performance with one such paper. However, to really make an impact and ensure we are doing good science, a thorough comparison should be made with different ways of achieving energy efficiency -- with adderNets being one of them. I agree with the Authors that pruning etc. can be applied to AdderNets also, but it's unclear how much pruning is possible with these architectures. 

While this paper is certainly an advance in the field of "AdderNets", I think it's critical to put it in the context of advances in energy-efficient computing, which is not done. Such a comparison would greatly strengthen the paper. I, therefore, encourage the authors to resubmit with this change. 

Decision: Accept (Poster)