Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: In this paper, the author combines channel projection, spatial projection, and the shortcut into a new architecture. The proposed is only validated on one dataset. 
Overall, the novelty of the proposed method is weak. The insufficient experiments can support the effectiveness of the proposed method. The paper is not well organized. Two many preliminaries are given, and some critical definitions are missing. 

strengths_and_weaknesses: Advantages:
1: A novel binary architecture is proposed and achieves SOTA performance.

Disadvantages:
1. The novelty is weak. The author combines the channel projection, spatial projection, and the shortcut into a new module. The core techniques channel projection, spatial projection, and the shortcut are existing techniques. The author modified them to be suitable for the binary vision MLP.
2. The improved performance is limited. Too many techniques (short-cut, BN+RPReLU, different MBB block settings a) are combined into one model, which only achieves 1.3% increase (71.4%->72.7%). 
3. The comparison experiment is not sufficient. Most of the compared methods are before 2020. The author should compare the proposed methods with some SOTA methods published in 2021 and 2022.
4. The proposed method is only validated on one dataset.
5. The paper is not well organized, and some important definitions are not given. For example, there are two pages for the preliminaries and Vision MLP Models before the proposed method. However, there are only two pages for the proposed method. The definition of two important methods Bi-MLP-S and Bi-MLP-M are missing.
6. The devised new framework is not useful. The core contribution combining special projection and channel projection doesn’t achieve a large increase (less than 0.8%) than the simple combination of special projection and channel projection (setting 1 in Table 3).

questions: 1. The advantage of ‘Ops’ is benefit from the downsampling layers. Are downsampling layers also added to the SOTA methods [17][2]? 
2. In the ablation study, the short-cut, BN+RPReLU, different MBB block settings achieve about 0.9%, 0.6%, and 0.8% increase, respectively. The overall framework only achieves 70% (Bi-MLP-S) and 72.2% (Bi-MLP-M) accuracy. The SOTA method (ReActNet) achieve 71.4% accuracy. So, which part brings the fundamental increase compared with SOTA methods?
3. What’s the meaning of Bi-MLP-S and Bi-MLP-M?
4. In page 7, the author mentioned that “The commonly used data-augmentation strategies such
 as Cut-Mix [33], Mixup [34] and Rand-Augment [5] are used. The first layer, the last layer and the downsampling layers are not binarized during training and inference.” Does the SOTA method also adopt the same data-augmentation strategy? The 1.3% increase is benefit from the data-augmentation strategy. The author is suggested to add the ablation study on the setting with and without data-augmentation strategy.

limitations: 1. The improved performance is limited. The proposed method only achieves 1.3% increase.
2. From Table3, we can see that setting 3 only achieves 0.8% increase than setting 1, which is a simple combination of special projection and channel projection. So I suspect that the main contribution of the combination of special projection and channel projection is not really useful.
3. The proposed method is only validated on one dataset.
4. The proposed method is the integration of existing techniques.
5. The author only gives the Ops performance with block setting from an experimental viewpoint. The author is suggested to add the theoretical analysis of inference performance with different block settings.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: In this paper, the authors first analyze the difficulty of binarizing MLP models due to the amount of FC layers (1x1 conv) that limit the ability for feature representation. Then, the authors propose mulit-branch binary MLP block to improve the accuracy and reduce the OPs compared to the stoa methods.
strengths_and_weaknesses: Strengths:

+Clear motivation and writing

The authors analyze the difficulty of binarizing MLP models and propose their solutions with MBB blocks. The writing is overall clear and straightforward.

+Strong experiment results

The authors conduct experiments based on MLP models and achieve better performance with less OPs.

Weaknesses:

-Missing related works

For the down-sampling layers proposed in this paper, some related works have been using such architecture like [1] but missing in the related work part.

[1] Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition, TPAMI 2015
questions: I don't have specific questions and suggestions for the authors. I tend to accept this paper but I don't have much experience on model quantization and binary networks. I probably change my rating after the rebuttal and discussion phase.
limitations: The authors didn't address their limitations and potential negative social impact.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper presents a binary MLP model. Specifically, it designs the binary block with multiple branches (MBB blocks) and uses the shortcut to encourage the information flow. For MBB blocks, the authors proposed to  form two different multi-branch blocks by using spatial projection and channel projection simultaneously. 
strengths_and_weaknesses: (+) The paper is well-written. The result shows it achieves even better accuracy than the state-of-the-art binary CNNs, which is impressive.

(-) I didn't see a strong motivation for using multi-branch blocks.

questions: 1. Any intuition about using the multi-branch blocks and the down-sampling block with multiple max-pooling branches?
2. What is the original size of the MLP network
3. Can the authors compare the FLOPs as well as the model size? Because MLP networks are computational efficient, but the #parameters might be much greater than CNNs.

limitations: The authors claimed the FC layers can be viewed as 1x1 Conv and thus is limited in representation ability. However, the number of channels in FC layers are usually much larger than 1x1 Convolutions used in the CNNs. In that case this claim doesn't quite hold.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes a new architecture for binarizing the vision MLP model and mitigates the gap between the FP32 model and the 1bit version. It designs a novel MBB block with a universal shortcut to strengthen the representation ability of the binary FC layer, and redesigns the downsampling layers to reduce the computational complexity while maintaining the classification performance. The experimental results on ImageNet dataset show that the performance of the proposed binary MLP model can exceed the state-of-the-art binary CNN model, and the ablation study demonstrates the effectiveness of each proposed module.
strengths_and_weaknesses: Strengths:
- The analysis of computational complexity and representation ability of binary conv layer and binary FC layer is inspiring. The author points out that the representation ability of a binary conv/FC layer is related to the kernel size and the input channel, which helps the design of the following MBB block. I think this analysis is also helpful for designing an effective binary CNN module.
- The use of multi-branch block is simple yet reasonable and effective. The authors uniformly distribute computation power on different dimensions, and substantially increase the representation ability while maintaining the computational complexity of binary FC layers, compared to binary conv layer with large kernel size.
- The experimental results show the effectiveness of the proposed architecture, and it outperforms the SOTA binary CNN model ReActNet. 

Weaknesses
- This paper should conduct an ablation study to show the overall benefit of modifying the architecture. For example, the accuracy gap between FP32 and the binary version of the original MLP and the modified MLP should be reported.
- When analyzing the representation ability and computational complexity, this paper assumes that the number of input channel and output channel of binary conv in CNN and binary FC in MLP are the same, is it true in reality?

questions: See weaknesses above. I suggest the author gives an example to prove that the assumption mentioned above is true in reality. Otherwise, it may influence the conclusion of the analysis.

Is 1x1 conv in the proposed downsampling block in Figure 3 binarized or remain FP32?

The font in Figure 2 should be unified.

limitations: Yes.
review_title: null

Meta-review: metareview: Four reviewers provided feedback on this paper. The authors provided a response to the reviews and I appreciate the authors' detailed comments and clarifications, specifically addressing each reviewer's comments/questions. The authors did not upload a revised version of the paper.

After the two discussion periods, three of the four reviewers suggest to accept the paper (with varying scores) while one reviewer (orUW) rated the paper as "borderline reject", so not strongly opposing acceptance. (Also, reviewer orUW chose to not engage in the discussion, nor did they acknowledge the authors' response, so their opinion should carry slightly less weight in the overall decision.) After considering the reviewers' and authors' comments, I believe that the paper should be accepted to NeurIPS.

Weaknesses include:
* The approach is only validated on one dataset.
* Concerns regarding missing related work. (Partially addressed in the response.)
* It would be great to see also concrete runtime or throughput numbers, not only OPs counts.

Strengths include:
* Novel approach to binary networks for image classification. Interesting discussion of binary networks in the context of CNN/MLP approaches. 
* New SOTA on ImageNet for binary networks (using fewer OPs).
* Experiments support claims.

I expect and hope that the authors will thoroughly address the reviewers' comments in the camera ready version of the paper.

Minor points (not affecting this decision, but potentially useful to authors when preparing the final revision):
* "sign function is non-differentiable almost everywhere" - it seems to me that the sign-function is actually differentiable almost everywhere, but has a zero gradient, maybe that is what is meant?
* typo: "Dowmsampling" (291)
confidence: Less certain
Decision: Accept