Binarized Convolutional Neural Networks with Channel Quadrupling and Smooth Downsampling

Authors: YOUNGWOOK KWON, Sejin Kwon, HyunJin Kim
Primary area: other topics in machine learning (i.e., none of the above)
Keywords: Binarized Convolutional Neural Networks, Image Classification, Image Segmentation, Mobile-Friendly Convolutional Neural Networks

Abstract:
This paper proposes novel binarized convolutional neural networks (BCNNs) named **QB-Net** and **QSB-Net**, specifically designed to **Q**uadruple the number of channels and incorporate a so-called **S**mooth downsampling in **B**CNNs for low-cost mobile environments. The proposed models combine FP32 depthwise separable (DS) convolutions with binarized $1 \times 1$ pointwise convolutions, offering reduced computational costs in the pointwise convolutions. To enhance the degraded performance of the above naive combination, the proposed models start with a small number of channels in shallow layers and expand them during downsampling by a factor of four, effectively managing model complexity in the downsampling. The proposed model structure maintains low computational costs in the shallow blocks and increases model complexity in the deep blocks, providing a wider dynamic range to manage information in the frequency domain. As a result, the proposed models overcome the limitations of existing BCNNs, delivering improved performance while reducing the total computational costs. For further performance enhancements, we propose a novel smooth downsampling with heightwise and widthwise sequential downsampling steps, doubling the number of channels at each step. Besides, we show that the channelwise self-attention (SE) is applicable with minimal additional computational costs in the proposed models. Besides, multiple binarized convolutions in the fully-connected (FC) layer reduce storage costs without requiring 8-bit quantized convolutions. Experimental results demonstrate the efficiency of the proposed models in terms of performance, computational costs, and inference latency on real hardware. Notably, the QSB-Net-Large with SE achieve 71.2\% Top-1 accuracy on ImageNet-1K and 69.2 mean intersection over union (mIoU) in the semantic segmentation on the PASCAL VOC dataset, outperforming other counterparts.

Decision: reject
Reviewer scores: 5, 6, 6, 5, 5

Reviews and discussion:
<thread>
creation date:
1730022959170

thread_type:
official_review

<thread_object>
creator: reviewer_d56q

<content>
summary:
This paper introduces two novel binarized convolutional neural network (BCNN) architectures, QB-Net and QSB-Net, which leverage CHANNEL QUADRUPLING and SMOOTH DOWNSAMPLING to optimize performance in resource-constrained settings. The proposed models integrate FP32 depthwise separable (DS) convolutions with binarized 1x1 pointwise convolutions. To manage model complexity effectively, both models use narrow channels in initial layers and expand channels by a factor of four during downsampling. Additionally, they introduce a smooth downsampling technique that progressively doubles the channel count at each stage. Further, the design incorporates channelwise self-attention (SE) with minimal computational overhead. Experimental results on ImageNet-1K and PASCAL VOC show that QB-Net and QSB-Net achieve superior performance among binarized networks.

strengths:
- The integration of channel quadrupling and smooth down-sampling within BCNNs represents a new approach that effectively balances computational efficiency with model performance.
- The reported accuracy on ImageNet-1K and PASCAL VOC datasets substantiates the efficacy of the proposed models.
- The latency measurements conducted on both the Raspberry Pi 4B and Samsung Exynos-9820 processor demonstrate the practical feasibility of the models in real-world applications.
- The paper offers comprehensive descriptions of the model architectures, experimental setups, and training protocols, enhancing transparency and supporting reproducibility.

weaknesses:
# Major Concern:
While the results presented are promising, the novelty of this paper appears somewhat limited. The model design heavily relies on a handcrafted architecture coupled with a brute-force search for optimizing the binary (or, more accurately, mixed-precision) network architecture. Given the current pace of advancements in the field, this approach may offer limited benefit to the broader community and does not fully align with the novelty standard typically expected at a top-tier machine learning conference.

# Minor Comments:

- The proposed models require over 500 epochs to converge, which is considerably longer than the conventional training time on the ImageNet dataset. To provide a more complete comparison with other binary and full-precision models, it would be beneficial to analyze the impact of these extended training requirements and the use of a teacher model on final model performance.
- Although the proposed models incorporate a range of established techniques through channel quadrupling and smooth down-sampling, the paper would benefit from additional ablation studies to clarify the individual contributions of these primary techniques to model performance.
- Additional visualizations of feature maps could be helpful in illustrating the effects of the proposed smooth down-sampling technique. This would also help clarify whether the observed performance improvements stem from the design itself or from added computational complexity.
- The paper’s primary focus is on image classification tasks, with limited experimental results on the VOC dataset for semantic segmentation. A broader range of experiments, particularly on larger and more diverse datasets (e.g., COCO detection or segmentation tasks), would help to more convincingly demonstrate the generalizability of the proposed model.
- It appears that more recent works on lightweight network design are missing from the comparisons in Table 3. Including these works would provide a more comprehensive context for evaluating the proposed model’s contributions.

questions:
- Could the paper provide a more extensive set of comparative experiments across a broader range of visual tasks and larger-scale datasets (e.g., COCO detection or segmentation tasks) to further validate the generalizability of the proposed model?
- Would it be possible to analyze and compare the impact of training costs on the final performance between the proposed models and existing binary or full-precision models? This would add valuable insight into the efficiency of the proposed approach.
- Could the authors include ablation studies to illustrate the individual contributions of channel quadrupling and smooth down-sampling to the overall model performance? This would clarify the significance of these techniques within the model design.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730175332875

thread_type:
official_review

<thread_object>
creator: reviewer_pzcm

<content>
summary:
This paper presents novel binarized convolutional neural networks (BCNNs), termed QB-Net and QSB-Net, which achieve state-of-the-art performance among BCNNs and are tested on edge devices. Experiments demonstrate the real-world deployment potential of QB-Net and QSB-Net.

strengths:
1.The deployments on RPi 4B and Exynos strengthen the practical relevance of this work.
2.The results across various datasets and tasks validate the effectiveness of QB-Net.

weaknesses:
1. The channel quadrupling technique is not novel; previous works, such as TResNet, have employed it. However, this work lacks a discussion on its application.

2. Figure 1(b) is unclear; moving the legend outside the figure may improve clarity.

3. The choice of model design requires a more thorough discussion.

4. The writing could be improved for clarity.

5. Additional tables in the Ablation Study section would be beneficial.

6. Could you clarify the impact of downsampling width first versus height first?

questions:
see weaknesses
</content>
</thread_object>

</thread>
<thread>
creation date:
1730454946704

thread_type:
official_review

<thread_object>
creator: reviewer_6hcy

<content>
summary:
This paper proposes novel binarized convolutional neural networks, which quadruple the number of channels and incorporate smooth downsampling. 
It is a mixed model with FP32 depthwise separable and binarized pointwise convolutions. 
It also starts with a few channels in shallow layers and expands them during downsampling by a factor of four, to improve the performance while reducing the total computational costs. 
Besides, a novel smooth downsampling, channel-wise self-attention and multiple binarized convolutions in the fully connected layer are also used. 
Experimental results demonstrate the efficiency of the proposed models in terms of performance, computational costs, and inference latency on real hardware on both ImageNet-1K classification and PASCAL VOC semantic segmentation.

strengths:
1. The motivation is reasonable. They hypothesize that information loss during downsampling can impact model performance, and develop a novel strategy for configuring channel expansion and downsampling in BCNNs.
   2. The result is comparable with the state-of-the-art.

weaknesses:
1. Typo: page 2 line 85, "signficant" should be "significant".
   2. Previous work [1] has proved the effectiveness of SE and its variant in BNNs, thus, it could not be regarded as a contribution. 
   3. The author claims they mainly focus on performance, computational costs, and inference latency. However, multiplying binarized convolutions in the fully-connected layer aims to reduce storage costs, and has no relation to this claim. 
   4. Why does the author quadruple the channel number? Could we set another expanding ratio like 3 or 5, instead of 4 used in this paper?

   [1] Training binary neural networks with real-to-binary convolutions. ICLR 2020

questions:
As listed in weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730996304645

thread_type:
official_review

<thread_object>
creator: reviewer_kyrr

<content>
summary:
This paper introduces QB-Net and QSB-Net, novel binarized convolutional neural network architectures specifically designed for mobile environments. It designed "channel quadrupling" that starts with fewer channels in shallow layers and expands them four-fold during downsampling, along with a "smooth downsampling" technique.

strengths:
1. Provides a fresh approach to BCNN design, moving beyond traditional methods.

2. The authors conduct real hardware testing on both Raspberry Pi 4B and Samsung Exynos processor, and the results demonstrate real-world applicability for mobile/edge devices. This is good.

weaknesses:
1. A major concern is that the performance improvement is marginal or null when compared with SoTA BNN. This method performs worse than PokeBNN [1], which was published three years ago. PokeBNN has a top1 acc of 70.5 on ImageNet, while it has fewer flops. 

2. Lacks deeper analysis of the relationship between model architecture and performance gains. I noticed that there are some performance degradation when trained from scratch. I am not sure if this performance gain from the designed modules themselves or from the training. 

[1] Pokebnn: A binary pursuit of lightweight accuracy, CVPR 2022

questions:
The proposed architectures combine FP32 depthwise separable convolutions with binarized pointwise convolutions, effectively reducing computational costs while maintaining performance through increased model complexity in deeper layers. It may improve the efficiency in terms of flops. **In system level, does the mix-precision design hurt the inference efficiency?**
</content>
</thread_object>

</thread>
<thread>
creation date:
1731056193949

thread_type:
official_review

<thread_object>
creator: reviewer_stxn

<content>
summary:
The paper introduces novel binarized convolutional neural networks (BCNNs), QB-Net and QSB-Net, designed to enhance performance in low-cost mobile environments by quadrupling the number of channels and incorporating smooth downsampling. The models combine FP32 depthwise separable (DS) convolutions with binarized 1×1 pointwise convolutions to reduce computational costs. The proposed structure starts with a small number of channels in shallow layers and expands them during downsampling, managing model complexity effectively. Experimental results demonstrate improved performance.

strengths:
1. The proposed QB-Net and QSB-Net models offer a new approach to channel expansion and downsampling in BCNNs, which is a  contribution to the field of model compression and quantization.
2. The paper demonstrates substantial performance improvements over existing BCNNs, which is a strong contribution for mobile and edge computing applications.
3. The paper provides a detailed explanation of the technical methods, including the new structure for channel quadrupling and smooth downsampling techniques.

weaknesses:
1. The paper may be seen as more of an engineering solution rather than a contribution to the foundational knowledge of the field.
2.  Although the paper claims reduced computational costs, the quadrupling of channels and the introduction of smooth downsampling could potentially increase the model's complexity, especially in deeper layers. This could lead to higher computational requirements that may offset the benefits of reduced pointwise convolution costs, especially the additional connections..
3. The abstract mentions improvements in inference latency on real hardware, but it does not discuss the energy efficiency of the models. For mobile environments, energy consumption is a critical factor, and the proposed models' impact on this aspect should be evaluated.

questions:
Mixed use of fp32 and binary value will increase the complexity of hardware design, how to solve it?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732285421702

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Request for Review of the Rebuttal

comment:
Dear Reviewers,

We have prepared this rebuttal by addressing your comments from two days ago. We understand you have many commitments and a busy schedule, but we hope to discuss our responses to your comments. If our rebuttal sufficiently addresses your concerns, we kindly ask you to reconsider your ratings.

Thank you.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732707278795

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Gentle Reminder for deadline of PDF modification

comment:
Dear Reviewers,

As the deadline for finalizing the revised version of the paper's PDF approaches, further modifications to the PDF are no longer possible. If reviewers raise additional discussions or concerns, we will do our best to address them thoroughly in the rebuttal. To respond to the concerns raised by the five reviewers, we have added several appendices and made revisions and additions to the figures and tables. Please review these sections, and **if you find that the concerns you raised have been adequately addressed, we kindly ask you to reconsider your rating.** Once again, we sincerely thank the reviewers for their valuable feedback.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732780499519

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Answer for Concerns from Reviewer d56q

comment:
Dear Reviewer d56q,

Thank you so much for reading our paper and providing various feedback. Regarding the concerns raised in the paper, our approach is as follows.
Firstly, we would like to acknowledge that there have been numerous approaches to binary neural networks (BNNs) in the literature. Many prior works on BNNs have optimized performance through **specific hyperparameters** or **employed customized sign functions,** leading to many publications in top-tier conferences. Additionally, we would like to point out that many of these studies only focus on image classification tasks.

However, we believe that such approaches can often result in **excessively long training times** for a single epoch, or performance improvements that are merely coincidental due to **repeated hyperparameter tuning**, making them **less practical.** 
Instead, we chose to focus on **structural changes to the model** and carefully examined whether **the design rules we applied could create an effective lightweight model.**

In particular, we explored the effects of adding more complexity during downsampling and analyzed the resulting changes in entropy and frequency domain characteristics. Through this, **we aimed to explain the phenomena observed and demonstrate that our model successfully achieves its fundamental goals of lightweight design and performance improvement.** 

We believed this makes our work a candidate for a top-tier conference.

Once again, we sincerely appreciate your feedback on our paper. With six days remaining, we welcome any additional comments or suggestions, as they would greatly help us further improve the quality of our work. Thank you again, Reviewer d56q.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733236526981

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Final Answer for Reviewers and ACs

comment:
Dear Reviewers and ACs,

  

Thank you for your valuable feedback during the discussion period. Your insights have helped us enhance the quality of our revised paper and refine the clarity of various ideas. However, we feel that some of the concerns raised may stem from misunderstandings or a lack of clarity in explaining our main contributions. As such, we would like to take this opportunity to summarize the remaining concerns and provide detailed explanations of our revisions. **Most of all, we ask the reviewers to consider not just the similarity in components but also the reasoning behind our architectural choices and the evidence we provide to support them.**
 

----------

  
* Key Remaining Concerns

  
1. Lack of novelty and perceived incremental nature of the work. The research is seen as focusing on the model architecture itself rather than offering insightful contributions.

2. Lack of clarity in application scenarios.

  

----------

  

**1. Lack of Novelty and Perceived Incremental Nature of the Work**

  

**Response:**

We kindly request the reviewers to evaluate not only the similarity in components but also the rationale behind our architectural decisions, along with the supporting efforts we have presented. **Notably, our work is rooted in a clear and motivated hypothesis:** that information loss during downsampling is a fundamental bottleneck in BCNNs. To address this, we introduced **channel quadrupling** and **advanced downsampling techniques.** These are not arbitrary design decisions but are grounded in the premise that **increasing representation capacity in deeper layers preserves richer feature representations, with reduced computational costs.**

  

This approach significantly enhances the model's ability to achieve **smoother entropy** and **broader dynamic range in frequency,** a crucial performance improvement substantiated through rigorous analysis. **We hope the reviewers will recognize this theoretical motivation and the systematic steps we took to validate it.**

  

----------
  

**2. Lack of Clarity in Application Scenarios**

  

**Response:**

We believe that **BCNNs are well-suited for high-performance microcontroller-level AI models,** where CPU-based bit-level parallelization is advantageous. Our target applications include platforms such as Raspberry Pi 4B and mobile devices, and we validated the performance of our model in these scenarios. Besides, previous BCNNs have only been evaluated on the Pascal VOC benchmark which is a performance limitation. However, we are the first BCNN model to evaluate **semantic segmentation and object detection on the COCO-Stuff benchmark**. Therefore, we demonstrate the potential applicability of the proposed model with improved results compared to models with FP32 precision.

  

----------

  

We deeply appreciate the reviewers' feedback, though we wish the discussion had provided more specific concerns to address or areas to elaborate on. This would have allowed us to offer even more detailed clarifications during the discussion period.

  

We kindly request the reviewers to reconsider the ratings in light of the above explanations, and we sincerely thank you again for your thoughtful feedback.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper proposed a binarized convolutional neural network utilizing channel quadrupling and smooth downsampling, and presented reasonable results to demonstrate the contribution of the paper. The reviewers had some concerns on the following: first, they feel the novelty is insufficient although the motivation seems intuitive. Second, they argue that the performance improvements over the existing strong competitors are marginal, which thus cannot fully reflect the advantages of the proposal. Lastly, the reviewers are not happy about the presentation quality because of typos and unclarity in the submission. Based on these comments, AC decided to recommend a rejection for this time.