Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper presents a method to improve upon binary neural network for image restoration (IR). It shows that batch normalization can help to obtain a good accuracy performance when binarizing IR models. Based on this observation, the authors introduce a basic binary convolution unit that benefits from batch norm properties using residual alignment. It has been shown that IR binary models outperform existing models across different tasks and datasets.

strength_and_weaknesses: Strengths:

-- The basic binary convolution unit is very simple and effective. IR models equipped with such units outperform existing works.

-- The binary IR models were tested on a wide range of tasks and datasets.

-- The paper is well-motivated, easy to read and well-structured.

Weaknesses:

-- The weight of technical novelty of this work is much lower than that of its empirical novelty. The proposed unit is obtained based on empirical experiments rather than a concrete mathematical foundation.

--  It is not very clear if the improvement comes from RA or RPReLU. What I am curious to see is an ablation study for BBCU-V4 without RA.

-- There is a small degradation in performance when binarizing body and upsampling parts in image super-resolution. I am wondering why it was not explored by prior works. If it was, then please compare.

clarity,_quality,_novelty_and_reproducibility: The paper is clear and its results can be reproduced. In terms of novelty, I believe its empirical novelty is more significant.
summary_of_the_review: In general, this paper introduces a simple and effective binary unit that further closes the gap between the binary model and its full-precision counterpart. The reasoning and development of the idea is based on empirical results, which its technical novelty is limited.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: In this paper, the authors study the components in binary convolution, such as residual connection, Batch Norm, activation function, and structure, for image restoration (IR) tasks. The aim is to conduct systematic analyses to explain each component’s role in binary convolution and discuss the pitfalls of such mechanisms.
strength_and_weaknesses: Strengths::
1) The motivation case study in section 1 is good.
2) The experimental section is good but can be improved based on the point mentioned in the weaknesses section.

Weaknesses::
The paper can be improved based on the following points.
1) The section 1 can be rearranged and written better. For example, analysis are provided in numerate bullet points but the paragraphing for each analysis is not consistent. Please rectify that for better readability.
2) It would be better to include some form of content table for better readability at the end of section 1 to describe the progression of the paper texts. Readability of the paper is not good and needs to be worked on.
3) Please specify each variables in Xjf⊗Wjf in section 3. Additionally, please specify what each avertible in the section is and why are they important. Please justify the design choices.
4) Section 4, though has a lot of experimental evaluation but lacks justification and reproducibility. Please explain why such evaluation are chosen and how can be reproduced successfully. Explanation is lacking.
clarity,_quality,_novelty_and_reproducibility: The paper lacks clarity and needs a lot of work to improve. Please note the weaknesses section in the review for further points on this. Novelty can't be determined with certainty as reproducibility of the work is lacking.
summary_of_the_review: 1) The section 1 can be rearranged and written better. For example, analysis are provided in numerate bullet points but the paragraphing for each analysis is not consistent. Please rectify that for better readability.
2) It would be better to include some form of content table for better readability at the end of section 1 to describe the progression of the paper texts. Readability of the paper is not good and needs to be worked on.
3) Please specify each variables in Xjf⊗Wjf in section 3. Additionally, please specify what each avertible in the section is and why are they important. Please justify the design choices.
4) Section 4, though has a lot of experimental evaluation but lacks justification and reproducibility. Please explain why such evaluation are chosen and how can be reproduced successfully. Explanation is lacking.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: Since the different properties in CNN and BNN, it hardly applies the experience of designing CNN to develop BNN. In this paper, the author rethinks and analyzes the actual function of components in binary convolution, such as residual connection, BatchNorm, activation function, and structure, for IR tasks. Based on their observations and analysis, they designed a general basic binary convolution unit for binarized image restoration networks. Extensive results can support their claimed contributions.
strength_and_weaknesses: Strengths: The paper is easy to follow and provides many insightful analyses, and the results are impressive and show promising potential for BNNs.
1. IR model compression is an important and urgent research direction, and BNN is a promising compression method. This paper tackles this problem and thus potentially can have a big impact.
2. This paper rethinks and analyzes the component of BNN. It is interesting and practical. The visualizations, experiments, and analyses are sound and interesting. 
3. The experiment results are impressive and general. The proposed model can significantly improve performance on different low-level vision tasks compared with SOTA methods. 
4. The ablation studies are insightful. It shows the binarization benefit of different parts and the performance of CNN and BNN to the residual connection, etc. 
5. The paper is well written and clearly organized.

Weakness： I have the following questions about the method and result details.
1. About the method and experiments, do you use the BBCU-V4 in Fig.2 to arm the IR networks? It is better to clarify this point.
2. In Tab.6, are the OPs computed based on 320×180 LQ images?  Besides, I hope the author provides OPs in Tab.4.  
3. Typos: In Fig.3 (b), the “BBCM-U“ seems not correct.
clarity,_quality,_novelty_and_reproducibility: This paper is well-organized and easy to follow. IR model compression is an important and urgent research direction and BNN is a promising compression method. It is worth exploring BNN on low-level vision. This article deeply analyzes the components of BNN and proposes a practical basic binary convolution unit for binary IR networks. It provides insightful and interesting analyses and conclusions in experiments to guide later BNN design and the results are impressive and general. Besides, the author provides demo codes to reproduce the results in the paper, which makes the paper more reliable and convincing.
summary_of_the_review: The paper is well written. The paper deeply analyzes the components of BNN and designs a BBCU. The ablation studies, like the visualization of feature distribution, demonstrate the effect of the method. The main comparison results with others are impressive and general. Overall, this paper is interesting, insightful, and practical.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper evaluated some essential design choices of existing binary neural networks for image restoration tasks. The authors further propose a network design and evaluate it on three different tasks. Experimental results show that the proposed binary network outperforms all the reference methods.
strength_and_weaknesses: **Strength**

- The paper is easy to follow.
- Experimental results seem to be strong. The proposed method outperforms all reference methods across all three tasks.

**Weaknesses**

- Limited novelty: The presented findings regarding residual connection, BN and the position of RPrelu are not supervising. Similar observations have already been reported and discussed in Bireal, Real2binaryNet and ReActNet papers. In terms of network design, there are only minor adaptions on top of ReActNet block design making the proposed BBCU block incremental. The current design lacks significant innovation.

- Current design choices are more based on empirical attempts and lack of refinement of optimization methods specified on IR tasks. For example, BNN-specific training methods such as two-stage training (real2binaryNet), progressive sign function ([1]), knowledge distillation training, etc., have not been verified in this work. However, these methods show significant improvement in the image classification task.

- If we want to demonstrate the practical effectiveness of BNN-based IR methods, it will be better to also compare with more compression methods, such as pruning-based methods and compact model designs. Also, is it possible to use a binary inference engine like larq or Dabnn to verify the real inference speed on Arm cpu? We know that the test results on hardware can sometimes be quite different from the theoretical computational complexity OPs. Arms cpu is also an important computing platform for IR tasks.

[1] Gradient matters: Designing binarized neural networks via enhanced information-flow
clarity,_quality,_novelty_and_reproducibility: No further concern in this category.
summary_of_the_review: The paper is well structured, and the experimental result seems to be strong. But, I still have several concerns regarding novelty, a more complete comparison, and adding inference speed evaluation.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This work looks to understand binary neural networks in the setting of modern CNNs with components such as normalization, skip connection, and various architectural settings. From this analysis, they further propose the basic binary convolution unit (BBCU) so as to outperform previous binary neural networks.

The reviewers overall enjoyed the experimental analysis (as well as myself) and convincing empirical results.

This paper had somewhat high ratings with the exception of one reviewer who gave a score of 3 (reject). Their concerns are effectively two: readability/clarity (which other reviewers didn't have much a problem with) and justification behind chosen experiments (not a significant deal). These are also easy adjustments, and unfortunately, the reviewer did not respond during the discussions or look at the rebuttal. I agree with the majority consensus.
justification_for_why_not_higher_score: Level of impact of work and technical novelty.
justification_for_why_not_lower_score: The reviewers overall enjoyed the experimental analysis (as well as myself) and convincing empirical results.
Decision: Accept: poster