BiMLP: Compact Binary Architectures for Vision Multi-Layer Perceptrons
Authors: Yixing Xu
Xinghao Chen
Yunhe Wang
Venue: NeurIPS 2022 Accept (2022)
Keywords: vision MLP
binary neural network
compact architecture
representation ability

Abstract: This paper studies the problem of designing compact binary architectures for vision multi-layer perceptrons (MLPs). We provide extensive analysis on the difficulty of binarizing vision MLPs and find that previous binarization methods perform poorly due to limited capacity of binary MLPs. In contrast with the traditional CNNs that utilizing convolutional operations with large kernel size, fully-connected (FC) layers in MLPs can be treated as convolutional layers with kernel size $1\times1$. Thus, the representation ability of the FC layers will be limited when being binarized, and places restrictions on the capability of spatial mixing and channel mixing on the intermediate features. To this end, we propose to improve the performance of binary MLP (BiMLP) model by enriching the representation ability of binary FC layers. We design a novel binary block that contains multiple branches to merge a series of outputs from the same stage, and also a universal shortcut connection that encourages the information flow from the previous stage. The downsampling layers are also carefully designed to reduce the computational complexity while maintaining the classification performance. Experimental results on benchmark dataset ImageNet-1k demonstrate the effectiveness of the proposed BiMLP models, which achieve state-of-the-art accuracy compared to prior binary CNNs.
The MindSpore code is available at \url{https://gitee.com/mindspore/models/tree/master/research/cv/BiMLP}.

BiMLP: Compact Binary Architectures for Vision Multi-Layer Perceptrons
Yixing Xu, Xinghao Chen, Yunhe Wang
Huawei Noah's 's Ark Lab (yixing.xi) xu, xinghao. chen, yunhe. vang] com
Abstract
This paper studies the problem of designing compact binary architectures for vision multi-layer perceptrons (MLPs). We provide extensive analysis on the difficulty of binarizing vision MLPs and find that previous binarization methods perform poorly due to limited capacity of binary MLPs. In contrast with the traditional CNNs that utilizing convolutional operations with large kernel size, fulnecconnected (FC) layers in MLPs can be treated as convolutional layers with kernel size when being binarized, and places restrictions on the capability of spatial mixing and channel mixing on the intermediate features. To this end, we propose to improve the performance of binary MLP (BiMLP) model by enriching the representation ability of binary FC layers. We design novel binary block that contains multiple branches to merge a series of outputs from the same stage, and also universal shortcut connection that encourages the information flow from the previous stage. The downsampling layers are also carefully designed to reduce the computational complexity while maintaining the classification performance. Experimental results benchmark dataset ImageNet- 1k demonstrate the effectiveness of the proposed BiMLP models, which achieve state-of-the-an accuracy compared to prior binary CNNs. The MindSpore code is available at nattpa://gitee http://aitee..commiinsppppr.. com/mindspore/ hooeel///rre///assrrr//eera/////
Thus, the representation ability of the FC layers will be limited
Introduction
Recent years have witness the boosting of convolutional neural networks (CNNs) on several computer vision (CV) applications, e.g., image recognition [19
38 14
11. object detection B55 34]
semantic segmentation [2) and low-level vision 21 However, the success CNN models highly depends their huge computational cost and massive parameters, which are not suitable to be directly applied to edge devices that have limited computational resources such as mobile phones, smart watches.
There are several model compression and acceleration methods to reduce the number of parameters and FLOPs of the original CNNs and derive portable model. For instance, knowledge distillation [16] aims to train a small student network with the help a cumbersome teacher network. Filter
48 49) pruning 29 15 42 41 methods sort the weights of the network based on their importance and throw away those who have negligible influence on the final performance. Model quantization methods reduce the original 32-bit floating point weights and activations into lower bits and tensor decomposition [| 46 methods express a large weight tensor as a sequence of elementary operations on a series of simpler tensors. Among them, binary CNN (53 12 45 an extreme of model quantization that uses only 1-bit for weights and activations. Compared to the original convolutional operation, binarized has 64x less FLOPs and 32x less memory consumption.
54
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Note that all of the existing binarization methods are applied on the convolutional operations in CNN models. However, multi-layer perceptron (MLP) model also shows its potential on CV tasks 4 3 T0 and matrix multiplication is computational friendly to GPUs and has advantage on inference time in reality. Compared to traditional CNNs that utilize convolutional operation with various kernel sizes, FC layers in MLP models can be treated as convolutions with kernel size 1 x 1, which suffer more from limited representation ability compared to the convolutional operations with larger kernel size when being binarized. In fact, directly binarizing MLP models with the existing methods show more severe performance degradation compared to the CNN models. For example, the Top-1 classification accuracy ImageNet will drop by more than 23% when binarizing CycleMLP [3] and Wave-MLP [40] with the method proposed in Dorefa-Nee [33] while the performance drop is only 17% for ResNet-18 [4] using 3 x kernel size for most of the convolutional layers and 13% for AlexNet 19 with larger kernel size (11 x 11 and 5 x 5).
Thus, in order to alleviate the problem men- tioned above, we introduce a novel binary block for MLP model that contains mul- tiple branches to merge a series of out- puts from the same stage of the network. Besides, a universal shortcut connection (Uni-shorteut) is designed to better trans- fer the knowledge from previous layers to the current layer. Compared to the origi- nal shortcut connection, the proposed Uni- shorteut can merge two features with dif- ferent shapes. By fusing the outputs of multiple layers derived from the same or different stages of the network, we show that the representation power of the binary MLP is drastically increased. The down- sampling layers are also modified reduce the computational complexity while main- taining the classification performance. Ex- perimental results on ImageNet-Il k dataset demonstrate that by using the proposed method, we achieve oftthor performance compared to other binary CNN models, as shown in Fig. [
75.0
(Oung)
70.0
RoActNet
NolusNet
Accuracy
67.5
65.0
RBML P (Ours)
62.5
MellusNet
RBMM ResNet-34
RENN
60.0
Net
BiRealNet
IR Nep
ReAct-Rtsstet
57.5
ReActNet
NesNei-18
Real28in
55.0
100
200
300
400
500
600
OPs (M)
Figure Accuracy vs. OPs of different binary models on ImageNet- 1 dataset.
We summarize our contributions for learning 1-bit vision MLPs as follows:
This paper points out the main difficulty of binarizing MLP models is that the representation ability of FC layers is worse than the convolutional layers in CNN models with kernel size larger than x
We introduce a mulit-branch binary MLP block (MBB block) with Uni-shorteut operation to unlock the representation power of binary MLP models and also modify the architecture downsampling layer to reduce the computational complexity.
The experimental results on ImageNet- Ik dataset show that the proposed BiMLP improves the top-1 accuracy by 1.3% with 12.1% less OPs compared to the state-of-the- ReActNet, which indicates the effectiveness of the proposed BiMLP models.
2 Preliminaries
2.1 Binarization in CNN models
Given the weights W Rexxexxxx where e; and c, are the number of input and output channels and & is the kernel size, and also the activation A E Re, xexx where h and w are the height and width of the input feature, common way of binarizing the weights and activations in CNN models to apply sign function on the 32-bi bit floating point inputs, i.e.,
(1) in which sign(-) (-) is an element-wise operation that outputs +1 when the input positive and -1 otherwise. Given the binarized weights and activations, the convolutional operation can be realized
We sign(W), As sign(A),

---

with binary operations only, i.e.:
Y W @ Ab:
(2)
in which represents binary convolutional operation that can be implemented with XNOR and POPCOUNT.
Note that the gradient of aforementioned sign function is zero almost everywhere and backward propagation is unable to be applied during training. Thus, the back- propppopaggtiion of sign function follows the straight through estimator (STE) rule:
aw Clipp OC ow, -1,1),
dc
(3)
where L is the loss function corresponding the specific task, e.g., cross-entropy loss for image classification. The Clip(,, -1 1) function clips the elements of the gradient into range [-1,
Most of the existing binary CNN methods focus improving the forward sign function and backward STE rule. For example, XNOR-Net В3] introduces channel- wise scale factors on the pure sign function and improves the performance of binary CNN. Dorefa-Net (53) uses only single scale factor to achieve similar performance. ReActNet [26] proposes RSign and RPreLU function for binary CNNs. DSQ [&] replaces the STE rule and uses the gradient of tanh function replace the gradient of sign function while BNN+ [6] introduces SignSwish function. Methods mentioned above are all applied CNN models and directly transfer them to MLP models results in severe performance degradation.
2.2 Vision MLP Models
Vision MLP models take patches of image (tokens) as input, and stack a sequence of channel-FC layers and spatial-FC layers for image recognition. Specifically, given the input X E Ruxd in which is the number of tokens and d is the number of channels in each token, the channel-FC layer fuses the channels in single token and generate d' channels, i.e.,
channel-FC(X, We) = X -W
(4)
where W. E Rdxd' the parameter matrix of the channel-FC layer, and represents the matrix multiplication operation. Channel-FC layer only aggregates information from different input channels, while lacking the communications between tokens. Thus, the spatial-FC layer is also introduced in MLP models 43) i.e.,
(5) where W2 E "x is the parameter of spatial-FC layer. In general, MLP models usually set d d' and n = n to pursuit efficiency on both feature representation and computation of the entire network.
spatial-FC(X, W.) = W.T X,
The spatial-FC layer mentioned above cannot deal with images with diverse shapes, thus is unable to be applied on downstream tasks such as image detection and image segmentation. In order to solve the problem, Cycle-MLP 3 retains the position of each token and introduces Cycle-FC layer to enlarge the receptive field of MI.P to cope with downstream tasks while maintaining the computational complexity. Specifically, given the input feature 7 E R H XCn, the output of Cycle-FC layer is:
C₂n
Cycle-FC(Z)
Σ
Zi48 Weyelee
(6)
C=U
in which Weyglle Cia XCout is the parameter matrix of the Cycle-FC layer, and dc(c) and 65(c) are defined as:
(c) (c mod Sn)
C 5,(c) (1 ( mod Sw) -
where Sh and Sw are predefined receptive fields. Besides Cycle-MLP, Wave-MLP [40] represents token wave function with amplitude and phase and proposes Wave-FC layer to solve the above problem. AS-MLP [22] avoids the restriction fixed input size by axially shifting channels of the feature map, and is able to obtain information flow from different axial directions.
Compared to the convolutional operation in CNN and self-attention operation in Transformer, matrix multiplication in MLP is computational friendly to GPUs and has specific advantage on inference time reality. Benefits from its simple architecture, MLP models can be generalized to various hardware. However, the cumbersome MLP model with a large number of parameters and FL.OPs limits its ability to apply to portable devices, which means is essential to derive a compact MI.P model. e.g. binary vision MLPs.
3

---

3 Binary Vision MLPs
In this section, we first analyze the difficulty of binarizing vision MLP through comparing the representation ability of binary features in FC layers and convolutional layers. We then propose novel binary architecture for vision MLP, which fuses the outputs from the same layer with multi- branch MIP block and those from different layers with universal shortcut connection to strengthen the representation ability of MLP.
3.1 Difficulty of Binarizing Vision MLP
Given the input feature F E R CCx and the convolutional kernel K CCux x KK in which K₁, and K are the height and width of the kernel, respectively. The computational complexity of the convolutional operation is:
O X Cout X Kn Kw x H x W).
After binarizing the weights and activations, the elements of the output feature map Y is derived from the following equation:
Cin
K
dz + K (c,i+ da) x (o,c, 2
Kn
Y(o,i,))
Σ
Σ
(9)
2
in which K and F are binarized kernel and input feature using Eq. whose elements selected from [+1,-13. According to Eq. 9 we can easily conclude that each element in Y is chosen from N+1 different values from the set 5, {-N, N.NN-N- - -2, 2,,.., ..., N - 2, N). in which N Cin K₁, K.- Specifically, we denote the number N as the representation ability of the binary FC layer.
In traditional CNN models such as ResNet [14] and VGGNet (88 convolutional operation with kernel size 3 x 3 occupies the network. Efficient-Net [39] uses convolution with kernel size 3 x and 5 5. while the recently proposed RepLKNet (7] expands the kernel size 31 x 31. Different from the CNN models mentioned above, FC layers in MLP model can be treated as convolutional
operation with kernel size x 1.
Therefore, compared to the convolutional layer with kernel size x k, FC layer with the same number of input channels and output channels has 1/k² computational complexity and 1/k representation ability after being binarized. Note that assuming CNN models and MLP models have the same number of input channels and output channels is reasonable. For example, Wave-MLP-S 40 with 30M parameters and 4.5G FLOPs has four stages with base channel 64- 64-128-320-512, while ResNet- 50 with 25.5M parameters and 4.1G FLOPs has four stages with base channel 64-128-256-512 which are roughly the same.
In order to achieve the same representation ability convolutional layer, the number of input channels of FC layer must be multiplied by 1:2.. Similarly, the number of output channels should also be scaled up in order to maintain the representation ability of the next FC layer. Hence, the computational complexity 1.2 times compared to the convolutional layer with kernel size k. which drastically reduce the advantage of binary neural network. To this end, we design a novel binary architecture for vision models (BiMLP) to enhance the representation ability while maintaining compact model complexity. A series of specific design are proposed to achieve this goal, which will be elaborated in the following sections.
3.2 BiMLP Architecture
To deal with the above problem, introduce a mulit-branch binary MLP block (MBB block) with Uni-shortcut operation to enhance the representation capacity of binary MLP models. We also design novel architecture of downsampling layer for binary MLPs to further reduce the computational complexity while maintaining the accuracy.
Multi-branch binary MLP block. To enhance the representation ability of binary vision MLPs with compact complexity, we propose a new binary MIP block containing multiple branches. Note that in MLP-Mixe (33,, ResMLP [44] and gMLP (25) MLP block mainly consists of two parts

---

Outputs
Outputs
RPRelL
FC (dim, 4xdim)
Spatial Bi-FC
BatchNorm
Channel Proj
GeLU
MBB Block 2
Spatil Bi-TC
Channel Bi
Spatial Bi-CC
Local Bi-FC
Uni-Shortcut
(dim, dim)
LayerNorm
BatchNorm
Channel Bi-FC
Channel
+
Channel Bi-FC
PATM (Wave-MLP)
Spatial Proj
3:7 PRTM
FC
74 PATM
MBB Block
Spati BI-MPP
Channel Bi-CC
Spatta BLMP
Spatial
CycleFC
LayerNorm
BatchNorm
RPfe LU
Channel Bi
BatchNorm
e.f. Wave-MLP
Global Bi-FC
Patch Embeddings
Patch Embeddings
Uni-Shortout
(a) Vision MLP
(b) BIMLP (Ours)
Figure 2: (a) Details of vision MLP blocks, e.g., Wave-MLP 40
(b) The proposed Multi-branch
binary MLP (MBB) blocks for BiMLP.
including the spatial projection part and the channel projection part. For CycleMLP B and Wave- MLP 40 the spatial projection is achieved by introducing some local FC operations (e.g. Cycle-FC, Wave-FC) which are applied the adjacent tokens cope with input images with different shapes. Meanwhile, the global FC layer (ordinary FC layer) used for channel mixing. Different from them, simultaneously introduce spatial projection and channel projection into single part, and form two diffcrent
binary MLP blocks (MBB blocks) as shown in Fig. 2
There are four different elements in the MBB blocks, i.e., the Spatial Binary FC, the Spatial Binary MLP, the Channel Binary FC and the Channel Binary MLP. Given the binarized input feature X the output of Spatial Binary FC is:
YSB_FC = RPR=LU((FC/BN((*)) U(X")),
(10)
in which RPReLU()) is the activation function introduced in ReActNet (6) LFC(.) is the local FC that fuses the spatial information which is realized with different form previous works. BN(.) is the
ordinary batch normalization, and U(-) is the proposed Uni-shorteut which will be introduced below
in Eq. [2] For Spatial Binary MLP, we utilize the PATM architecture introduced in Wave-MLP 40
Other forms such CycleMLP architecture in 3 and axially shifting architecture AS-MLP can be used as replacement.
The Channel Binary FC is defined as:
YCB = RPReLU(GFC(BN(X")) + +UX)));
(II)
in which GFC(-) is the original global FC layer that mixes the information between different channels. Finally, the Channel Binary MLP is the stack of multiple Channel Binary FCs to strengthen the ability of channel mixing.
Given the four different elements introduced above, the first MBB block uses two Spatial Binary MI.Ps focusing on the height and width of the spatial dimension, and a Channel Binary FC. The second MBB block uses two Spatial Binary FC and a Channel Binary MLP. this way, the channel projection and spatial projection used multiple times, while at the same time the first block has stronger ability for spatial mixing and the second block is good at mixing the channel information. Finally, all the layer normalizations replaced with batch normalizations, as shown in the ablation study in Tab.
With the above architectures, the representation ability can be recovered with less computational complexity by using multiple branches compared with directly expanding the input and output chan- nels. Generally, k2 branches are needed to get the same representation ability, and the computational complexity is the same compared to the convolutional layer with kernel size k. Note that as the number of branches increases, both the representation ability and the computational complexity
5

---

H W Ci
conv stride=1
H Can.
H X W Cout
x 3 conv
2 maxpool
3 3 maxpool stride=2
x maxpool stride=2
7 7 maxpool stride=2
stride=2
stride=2
H W 2 2
x Cant
H
W 2
x Cour 2
Figure 3: Left: The original downsampling layer used in 32-bit floating point MLP models. Right: The proposed downsampling block for BiMLP.
will increase. Thus, we seek for a balance between the effectiveness and the efficiency. In the following experiments, we use only 3 branches as mentioned above and show that it is enough to obtain competitive results Tab.
Recall that the proposed MBB blocks fuse the information from the same layer. The previous works of binary CNNs show that combining the outputs from different layers is important for the information flow. However, the number of output channels and input channels are usually different in MLP models and the original shortcut connection cannot be directly applied. Since the input channels Cin. are usually multiple times of the output channels Cout (or vise versa), i.e. Cin = nout (or Cout = nCC.). n N+, Therefore, we propose a universal shortcut (Uni-shortcut) to cope with the following two different situations.
Uni-shortcut.
Given the input feature X E CC CCx we have:
xxxW and FC layer with Gi and Cout input and output channels,
Cin
=
Cin, Gin nCoot
n
U(x"=
(12)
concat
x
dim 2), Cout BCCn
where the input is averaged based on the channel dimension when Cin TCoout and the input is repeated n times and concatenated together to fit the output dimension when Cont = 7Cin-
Downsampling block. The downsampling layers are not binarized during training and inference for MLP models, otherwise the performance will drastically decrease, as also discussed in prior binary CNNs (27 266] [3]. This brings the problem that the FLOPs of downsampling layers occupy the whole network since the x convolution with stride 2 a common way simultancously reducing the spatial size and changing the number of channels of the feature maps. Thus, similar to [[3]. we separately handle the spatial and channel dimension by using 1 x 1 convolution with stride (FC layer), and maxpooling with diverse kernel size to replace the original convolution, as shown in Fig. 3 By this way, the computational complexity is significantly reduced while classification performance is maintained.
Experiments
In this section, we first evaluate the effectiveness of the proposed method on ImageNet dataset. The experimental settings include the dataset statistics, details of the network architectures and the training strategy is introduced Sec. [.]] In Sec. 4.2] we compare our method with a series state-of-the-an binary CNN models in terms of accuracy and FLOPs. The ablation study each part of the proposed
method is conducted in Sec.
4.1 Experimental Settings
ImageNet dataset. The benchmark dataset ImageNet- 1k (36 contains over 1.2M training images and 50k validation images from 1,000 different categories. This is the most commonly used image classification dataset in other binary CNN methods.

---

Table I: Details of different architectures of BiMLP. The parameter 'dim" indicates the dimension of features, and 'ratio' is the expand ratio of the mid layer of MLP. FLOPs is calculated on the layers that are not binarized, BOPs is calculated on binarized layer and OPs = BOPs / 64 + FLOPs. The computational complexity is calculated based on the input shape 224 x 224.
Output size
BiMLP-S
BiMLP-M
dim = 64
dim = 64
Stage 1
x
ratio = 4
2
ratio = 4
2
dim 128
dim = 128
Stage 2
g x w
2
3
ratio = 4
ratio = 4
dim = 320
dim = 320
Stage 3
16 x
ratio = 4
4
ratio =
10
dim = 512
dim = 512
Stage 4
x
ratio 4
ratio 4
FL.OPs (x10")
1.21
1.21
BOPs (x10")
2.25
4.32
OPs (x10*)
1.56
1.88
Network architecture. We utilize the architecture proposed in 40) and replace the MLP block with the proposed MBB block with Uni-shorteut (Fig. 2b)) as well as the replacement downsampling blocks (Fig. 3). Details of the architectures of BiMLP models are specified in Tab. !
Training details. Following prior methods 27
26 30 we use two step training strategy. In the
first training step, we use the full precision MLP model as the teacher model and the network with the same architecture but binary activations the student model. A knowledge distillation loss is used to facilitate the training of student network:
L Laiy v.) +11 (1-10) a) Σ Ny..
(13)
in which (-)(-) and (-) (-) are the KL divergence and cross-entropy loss, y. and y: the output probability the student and teacher model, Yut is the one-hot ground-trute tounnd-ruthhhu probability and a is the trade-off hyper-parameter (a = 0.9 in the following experiments). For the second training step, the full precision ResNet-34 is used as the teacher network. The binarized MLP model (both weights and activations) is used the student model and the corresponding weights are initialized by the training results from the first step.
(28 optimizer with In both steps, the student models are trained for 300 epochs using Adam W
momentum of 0.9 and weight decay of 0.05. We start with the learning rate of 10 and a cosine learning rate decay scheduler is used during training. We use NVIDIA V100 GPUs with total batchsize of 1024 to train the model with Mindspore [17). The commonly used aata-augmentation
strategies such as Cut-Mix (5I) Mixup [52] and Rand-Augment (5) are used. The first layer, the last layer and the downsampling layers are not binarized during training and inference.
4.2 Experimental Results on ImageNet
In this section, we compare the proposed model with the binary CNN models derived from other
thae--of methods on the ImageNet-IK 1k dataset, including Dorefa-Net [53], XNOR-Net [33].
ABCNet 24) Bireal-Net (27). Real2bin (30), MeliusNet (2), ReActNet 26]. etc. As shown in Tab. 2
the proposed BiMLP models achieve competitive performance compared to the state-of of-therrtt binary CNN models. We can see that BiMLP-S achieves 70.0% top-1 accuracy and 89, 89.6% top-5 accuracy with 0.156G OPs, which surpasses the most recent methods such as MeliusNet-42 by 0.8% and 1.3% with less than half OPs, and is comparable to ReActNet-B. Meanwhile, BiMLP-M achieves 72.7% top-1 accuracy and 91. 5% top-5 accuracy with only 0. 18888 OPs, which has 12. 2.19 % less OPs than ReActNet-C and 1.3% % better on top-1 accuracy. The above results show the superiority of the proposed BiMLP.
7

---

Table 2: Experimental results ImageNet-Ik using different binary CNN models and the proposed BiMLP. Bit-width (W/A) indicates the bit length of weights and activations. FLOPs calculated on the full-precision layers, BOPs is calculated binarized layers and OPs BOPs / 64 FLOPs.
Bit-width (W/A)
FLOPs (x10*)
BOPs (x10%)
OPs (x (x10*)
Top-I Acc (%)
Top-5 Acc (%)
Methods
XNOR-Net
1/1
1.41
1.70
1.67
$1.2
73.2
Dorefa-Net
S3
1/1
52.5
67.7
ABCNeI 2
1/1
42.7
67.6
Bireal-Net-
27
1/1
1.39
1,68
1.63
56.4
79.5
Bireal-Net-33
1/1
1.39
3.53
1.93
62.2
83.9
UAD-BNN-I 18
1/1
57.2
80,2
UAD-BNN-34
18]
1/1
62.8
84.5
RBNN-18 23
1/1
59.9
81.9
RBNN-34 23
1/1
63.1
84.4
Real2bin
1.56
1.83
65.4
86.2
ReCU (5
1/1
66.4
86.5
FDA-BNN 47
1/1
66.0
86.4
LCR-BNN- 18
37
1/1
59.6
81.6
LCR-BNN 34
37
1/1
63.5
84.6
LCR-BNN -cetctt
1/1
69.8
85.7
Bi-half-18 20
1/1
60.4
82.9
Bi-half-34 20
1/1
64.2
85.4
AdaBin-18 455
1/1
1.41
1.69
1.67
66.4
86.5
AdaBin-ReAco
70.4
AdaBin-Melius59
(45
1/1
5.27
71.6
MeliusNet-22 2
1/1
1.35
4.62
2.08
63.6
84.7
MeliusNet-29
1/1
1.29
5.47
2.14
65.8
86.2
MeliusNet-42
1/1
1.74
9.69
3.25
69.2
88.3
McliusNet-59
1/1
2.45
18.3
5.25
71.0
89.7
ReActNet-B 26
1/1
0.44
4.69
1.63
70.1
ReActNet-C 26
1/1
1.40
4.69
2.14
71.4
BiMLP-S (Ours)
1/1
1.21
2.25
1.56
70.0
89.6
BiMLP-M (Ours)
1/1
1.21
4.32
1.88
72.7
91.1
Table 3: Experimental results of using different branches in MBB blocks. The experiments conducted using BiMLP-S model on ImageNet- 1k dataset, and setting used as the baseline architecture which is the same Fig. (b)
MBB Block
MBB Block 2
OPs (x10*)
Top-1 Acc
Sctting
#S
#C'
#S2
#Ca
(%)
4
2
1.60
69.2
2
0
2
0
1.53
68.9
3
2
2
1
1.56
70.0
4
4
2
1.86
70.8
2
1
1.78
70.5
4.3
Ablation Studies
In this section, we demonstrate the effectiveness of each part of the proposed method by conducting a series of ablation studies.
Different branches MBB blocks. Firstly, we show the experimental results of using different branches in Tab. 3 The #S,, indicates the number of Spatial Binary FC (MLP) in the MBB block and #C; is the number of Channel Binary FC (MLP) in the corresponding block. Note that we fuse the information along height and width of the feature map with different Spatial Binary FC (MLP), thus the #S, should be even numbers. During the ablation study we keep (#S) (552/(#C, + #C2) = that the ability of fusing information of different dimensions is roughly the same.
The experiments are conducted on BiMI.P- model. As shown in Tab. 3 when the MBB blocks focus on only one dimension in setting and 2, the top-1 accuracy decreases from 70.0% to 69.2%

---

Table 4: Different forms of normalization layer and activation function. conducted using BiMLP-S model on ImageNet-1K dataset.
The experiments
Normalization
LN
LN
LN
LN
BN
BN
BN
BN
Activation
GeLU
ReLU
PReLU
RPReLU
GeLU
ReLU
PReLU
RPReLU
Top-1 Acc (%)
68.2
67.8
68.4
69.2
68.5
68.1
69.4
70.0
Table 5: Experimental results of using differ- ent form of residual connections. The exper- iments are conducted ImageNet-Ik dataset with BiMLP-S model.
Table 6: Results of using different downsampling layers. 'Original' indicates 3 3 convolution with stride 2, and 'proposed" means the proposed
downsampling block as shown Fig. right.
Top-1 Ace (%)
OPs (x10*)
Top-1 Acc (%)
Setting
w/o shortcut 68.3
w/ Uni-shorteut
70.0
Original
2.65
70.3
w/ shortcut
69.1
Proposed
1.56
70.0
and 68.9%, which shows that mixing the information from different dimensions a single block is important. In setting 4 and 5, more branches used in each block compared setting 3 (baseline), and only improves the accuracy by 0.8% and 0.5%, while increases 19.2% and 14. % OPs. Thus, we use setting for all the experiments considering the balance between the effectiveness and efficiency.
Normalization and activation. Different forms of normalization layer and activation function are compared. The candidates of normalization layer include Layer normalization (LN) and batch normalization (BN), and the candidates of activation functions are GeLU, ReLU, PReLU and RPReLU. As the results shown in Tab. the combination of BN and RPReLU achieves the best accuracy, and is applied to all other experiments.
Shorteut connection. We also study the effectiveness of the proposed Uni-shorteut. Three different setting are used in this ablation study. The first is using the MBB block without any residual connection. The second is using the proposed Uni-shorteut in MBB blocks. The third is using the traditional shortcut MBB blocks. this circumstances, only those input-output pairs who have same shape will be connected.
As shown in Tab. [5] compared the MBB block without using any residual connection and using the traditional shortcut, Uni-shortcut improves the top-1 accuracy by 1.7% and 0.9% which shows the priority of the proposed method.
Downsampling layer. We compare the proposed downsampling block using the FC layer together with multi-branch maxpooling to the original downsampling layer using 3 convolution with stride 2 in Tab. 6 The result shows that we can significantly reduce the computational complexity from 0.265G to 0.156G (41.1% decrease) with only 0.3% drop of top-1 classification accuracy.
Hyper-parameters. As shown in Eq. 13 a is used as aaper-parameter to balance the loss function between learning from the output probability of teacher model and learning from the ground-trute
Table Results of using different binarization methods on ImageNet- 1 dataset. The "FP32" model indicates the Wave-MLP model, and "I- bit' model uses sign function and STE rule binarize the FP32 model.
70.0
69.6
69.4
Model
Top-1 Acc(%)
69.2
FP32 1-bit
80.
69.0
55.3
+ Dorefa-Net 553
58.4
68.6
+ XNOR-Ne
33
59.1
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
+ ReActNet 26
63.2
Ours
70.0
Figure 4: Top-1 1 acc vs. on ImageNet val set.

---

one-hot label. We report the accuracy on ImageNet- Ik validation set in Fig. 4 and show that a 0.9 yields the best result.
Comparisons of different binarization methods. We compare the proposed method to other binarization methods proposed for CNN models in Tab. 7 The 1-bit model is to directly binarize the FP32 Wave-MLP model using the sign function and the STE rule. The experimental results show that our method can significantly outperform other binarization methods that are directly applied to MLP model.
Overall benefit of modifying the architecture. We con- duct an ablation study to show the overall benefit of mod- ifying the architecture. The accuracy gaps between FP32 and the binary version of the original MI.P and the modi- fied MLP are reported in Tab. 8 We can that the FP32 variant the modified model achieves roughly the same performance with original FP32 model. Meanwhile, our proposed BiMLP architecture outperforms the 1bit Wave- MLP with large margin, which justifies the effectiveness of the proposed binary vision MLP architecture.
Table Comparisons of FP32 and bi- nary variants for original and our pro- posed architectures on ImageNet- lk.
Model
Wave-MLP
BiMLP
FP32 1-bit
80.1
79.9
63.2
70.0
Conclusion
In this paper, we propose MLP models. We point out that compared to binarizing the convolutional operation with large kernel size in CNN models, binarizing the FC layers yields MLP model with poor representation ability. Simply increase the number of channels will fix this problem, but the computational complexity is quadratically increased. Thus, we propose a novel multi-branch binary MLP block (MBB block) with universal shortcut (Uni-shorteut) that can recover the representation ability while maintaining the computational complexity. We also redesign the downsampling layer that can significantly reduce the OPs of the binary MLP model while at the same time keep the performance roughly unchanged. The experimental results on ImageNet- 1 dataset demonstrate the effectiveness of the proposed method, and we achieve comparable performance with the state-of-the-an binary CNN models.
paradigm with specific architecture design for binarizing the vision
Acknowledgments
We gratefully acknowledge the support of MindSpore, CANN (Compute Architecture for Neural Networks) and Ascend AI Processor used for this research.
References
[1] Davide Bacciu and Danilo P Mandic. Tensor decompositions in deep learning. arXiv preprint MrXiv:2002.11833 2020.
[2] Joseph Bethge, Christian Bartz, Haojin Yang. Ying Chen, Christoph Meinel. MeliusNet: Can binary neural networks achieve mobilenet-level accuracy? In WACV, 2021.
[3] Shoufa Chen, Enze Xie. Chongjian Ge, Ding Liang. and Ping Luo. CycleMLP: A mip-like architecture for dense prediction. In ICLR, 2022.
[4] Brian Chmiel, Ron Banner, Gil Shomron, Yury Nahshan, Alex Bronstein, Uri Weiser, et al. Robust guantization: One model rule them all. NewrIPS, 33:5308 5317, 2020.
[5] Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoe Le. Randaugment: Practical automated data augmentation with a reduced search space. In CVPR Workshops, pages 702-703, 2020.
[6] Sajad Darabi, Mouloud Belbahri, Matthieu Courbariaux, and Vahid Partovi Nia. Bun+: Improved binary network training. 2018
Xiaohan Ding. Xiangyu Zhang, Yizhuang Zhou, Jungong Han, Guiguang Ding, and Jian Sun. Scaling up your kemels to 31x31: Revisiting large kernel design in enns. In CVPR, 2022.
[8] Ruihao Gong, Xianglong Liu, Shenghu Jiang. Tianxiang Li, Peng Hu, Jiazhen Lin, Fengwei Yu, and Junjie Yan. Differentiable soft quantization: Bridging pill-preeision and low-bit neural networks. In ICCV, 2019.
10

---

[9] Jianyuan Guo. Kai Han, Han Wu, Yehui Tang, Xinghao Chen, Yunhe Wang. and Chang Xu. Cmt: Convolutional neural networks meet vision transformers. In CVPR, pages 12175-12185, 2022.
[10] Jianyuan Guo, Yehui Tang. Kai Han, Xinghao Chen, Han Wu, Chao Xu, Chang Xu, and Yunhe Wang. Hire-mlp: Vision mip via hierarchical rearrangement. In CVPR, pages 826 836, 2022.
[11] Kai Han, Yunhe Wang. Qi Tian, Jianyuan Guo. Chunjing Xu, and Chang Xu. Ghostnet: More features from cheap operations. In CVPR, pages 1580-1589, 2020.
[12] Kai Han, Yunhe Wang. Yixing Xu, Chunjing Xu, Enhua Wu, and Chang Xu. Training binary neural networks through learning with noisy supervision. In ICML, pages 4017-4026. PMLR, 2020.
[13] Kaiming Hc, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Spatial pyramid pooling in deep convolutional networks for visual recognition. JEEE T-PAMI, 37(9):100411166 2015.
[14] Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, pages 770-778, 2016.
[15] Yang He, Guoliang Kang, Xuanyi Dong, Yanwei Fu, and Yi Yang. Soft filter pruning for accelerating deep convolutional neural networks. In IJCAI, 2018.
[16] Geoffrey Hinton, Oriol Vinyals, Jeff Dean, et al. Distilling the knowledge a neural network. arXiv preprint 1rXiv:1503.02531, 2(7), 2015.
[17] Huawei. Mindspore. https: http:///tttttt////////////papaa wwwp://wwwwwww..........
cn/ 2020.
[18] Hyungjun Kim, Jihoon Park, Changhun Lee, and Jae-Joon Kim. Improving accuracy of binary neural networks using unbalanced activation distribution. In CVPR, pages 7862-7871, 2021.
[19] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convolutional neural networks. NeurlPS, 25, 2012.
[20] Yunqiang Li, Silvia-Laura Pintea, and Jan C van Gemert. Equal bits: Enforcing equally distributed binary network weights. In AAAI, volume 36, pages 1491-1499, 2022.
[21] Zhen Li. Jinglei Yang. Zheng Liu, Xiaomin Yang, Gwanggil Jeon, and Wei Wu. Feedback network for image super-resolution. In CVPR, pages 3867-3876, 2019.
[22] Dongze Lian, Zehao Yu, Xing Sun, and Shenghua Gao. AS-MLP: An axial shifted mlp architecture for vision. In ICLR, 2022.
[23] Mingbao Lin, Rongrong Ji, Zihan Xu, Baochang Zhang, Yan Wang. Yongjian Wu, Feiyue Huang, and Chia-Wen Lin. Rotated binary neural network. NeurlPS, 33,7474748, 2020.
[24] Xiaofan Lin, Cong Zhao, and Wei Pan. Towards accurate binary convolutional neural network. NeurIPS, 30, 2017.
[25] Hanxiao Liu, Zihang Dai, David So, and Quoe V Le. Pay attention to mlps. In NeurlPS, volume 34, pages 9204-9215, 2021.
[26] Zechun Liu, Zhiqiang Shen, Marios Savvides, and Kwang- Ting Cheng. Reactnet: Towards precise binary neural network with generalized activation functions. In ECCV, pages 143-159. Springer, 2020.
[27] Zechun Liu, Baoyuan Wu, Wenhan Luo, Xin Yang, Wei Liu, and Kwang-Ting Ting Cheng. Bi-real net: Enhancing the performance of 1-bit cnns with improved representational capability and advanced training algorithm. In ECCV, pages 722-737, 2018
[28] Ilya Loshchilov and Frank Hutter. arxiv:1711.05100, 2017.
Decoupled weight decay regularization. arXiv preprint
[29] Jian-Hao Luo, Jianxin Wu, and Weiyao Lin. Thinet: A filter level pruning method for deep neural network compression. In ICCV, pages 8058-5066, 2017.
[30] Brais Martinez, Jing Yang, Adrian Bulat, and Georgios Tzimiropoulos. Training binary neural networks with -cil-ro-binary convolutions. arXiv preprint arXiv:2003. /1535, 2020.
[31] Ying Nie, Kai Han, and Yunhe Wang. Mut adaptive distillation for binary neural networks. In BMVC, 2021.

---

[32] Hyeonwoo Noh, Seunghoon Hong, and Bohyung Han. Learning deconvolution network for semantic segmentation. In ICCV. pages 1520-1528, 2015.
[33] Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xnor-net: Imagenet classifica-
tion using binary convolutional neural networks. In ECCV, pages 525-542. Springer, 2016.
[34] Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection, In CVPR. pages 779-788, 2016.
[35] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn; Towards real-time object detection with region proposal networks. NewrlPS, 28, 2015.
[36] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang. Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. IJV, 115(3):211-22)) 1-252, 2015.
[37] Yuzhang Shang, Dan Xu, Bin Duan, Ziliang Zong. Liqiang Nie, and Yan Yan. Lipschitz continuity retained binary neural network. ECCV, 2022.
[38] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recogni- tion. arXiv preprint 1409v 2014.
[39] Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In ICML, pages 6105-6114. 14. PMLR, 2019.
[40] Yehui Tang, Kai Han, Jianyuan Guo, Chang Xu. Yanxi Li, Chao Xu, and Yunhe Wang, An image patch is wave: Phase- aware vision mlp. In CVPR, 2022.
[41] Yehui Tang, Yunhe Wang. Yixing Xu, Yiping Deng, Chao Xu, Dacheng Tao, and Chang Xu. Manifold regularized dynamic network pruning. In CVPR, pages 5018-5028, 2021.
[42] Yehui Tang, Yunhe Wang, Yixing Xu, Dacheng Tao, Chunjing Xu, Chao Xu, and Chang Xu. Scop: Scientific control for reliable neural network pruning. NeurlPS, 13:10936-10947, 2020.
[43] Ilya O Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung. Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, et al. Mlp-mixer An all-mlp architecture for vision. NeurIPS, 34, 2021.
[44] Hugo Touvron, Piotr Bojanowski, Mathilde Caron, Mathhiuu Cord, Alaaeldin El-Nyuby, Edouard Grave, Gautier Izacard, Armand Joulin, Gabriel Synnaeve, Jakob Verbeek, et al. Resmlp: Feedforward networks for image classification with data-efficien training. arXiv preprint (:Xiv:2105.034:0 2021.
[45] Zhijun Tu, Xinghao Chen, Pengju Ren, and Yunhe Wang. AdaBin: Improving binary neural networks with adaptive binary sets, In ECCV, 2022.
[46] Yinan Wang, Weihong "Grace" Guo, and Xiaowei Yue. Tensor decomposition to compress convolutional layers in deep learning. IISE Transactions, 54(5):481 415, 2022.
[47] Yixing Xu, Kai Han, Chang Xu, Yehui Tang, Chunjing Xu, and Yunhe Wang. Learning frequency domain approximation for binary neural networks. NeurlPS, 34:25553-2556,,222 2021.
[48] Yixing Xu, Yunhe Wang, Hanting Chen, Kai Han, Chunjing Xu, Dacheng Tao, and Chang Xu. Positive- unlabeled compression the cloud. NeurlPS, 32, 2019.
[49] Yixing Xu, Chang Xu, Xinghao Chen, Wei Zhang, Chunjing Xu, and Yunhe Wang. Kernel based progressive distillation for adder neural networks. NewrIPS, 33:12322-12333 2020.
[50] Zihan Xu, Mingbao Lin, Jianzhuang Liu, Jie Chen, Ling Shao, Yue Gao, Yonghong Tian, and Rongrong Ji. Recu: Reviving the dead weights in binary neural networks. ICCV. pages 3198-5208, 2021.
[51] Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In ICCV, 2019.
[52] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In ICLR, 2018
[53] Shuchang Zhou, Yuxin Wu, Zekun Ni, Xinyu Zhou, He Wen, and Yuheng Zou. Dorefa-net: Training low bitwidth convolutional neural networks with low bitwidth gradients. arXiv preprint arXiv: 1606.06 60. 2016.
[54] Yiren Zhou, Seyed- Moyed-Mohsen Goosavi-Dezfooll Ngai- Mani-Man&oe Cheung, and Pascal Frossard, Adaptive quanti- zation for deep neural network. In AAAI, volume 32, 2018
12

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits. hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g.. with respect the random seed after running experi- ments multiple times)? [No]
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [N/A]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're rou're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
13


Reviews:
[{"review_id": "Smfma6uKX7F", "reviewer": "Reviewer_orUW", "summary": "In this paper, the author combines channel projection, spatial projection, and the shortcut into a new architecture. The proposed is only validated on one dataset. \nOverall, the novelty of the proposed method is weak. The insufficient experiments can support the effectiveness of the proposed method. The paper is not well organized. Two many preliminaries are given, and some critical definitions are missing. \n", "questions": "1. The advantage of ‘Ops’ is benefit from the downsampling layers. Are downsampling layers also added to the SOTA methods [17][2]? \n2. In the ablation study, the short-cut, BN+RPReLU, different MBB block settings achieve about 0.9%, 0.6%, and 0.8% increase, respectively. The overall framework only achieves 70% (Bi-MLP-S) and 72.2% (Bi-MLP-M) accuracy. The SOTA method (ReActNet) achieve 71.4% accuracy. So, which part brings the fundamental increase compared with SOTA methods?\n3. What’s the meaning of Bi-MLP-S and Bi-MLP-M?\n4. In page 7, the author mentioned that “The commonly used data-augmentation strategies such\n as Cut-Mix [33], Mixup [34] and Rand-Augment [5] are used. The first layer, the last layer and the downsampling layers are not binarized during training and inference.” Does the SOTA method also adopt the same data-augmentation strategy? The 1.3% increase is benefit from the data-augmentation strategy. The author is suggested to add the ablation study on the setting with and without data-augmentation strategy.\n", "limitations": "1. The improved performance is limited. The proposed method only achieves 1.3% increase.\n2. From Table3, we can see that setting 3 only achieves 0.8% increase than setting 1, which is a simple combination of special projection and channel projection. So I suspect that the main contribution of the combination of special projection and channel projection is not really useful.\n3. The proposed method is only validated on one dataset.\n4. The proposed method is the integration of existing techniques.\n5. The author only gives the Ops performance with block setting from an experimental viewpoint. The author is suggested to add the theoretical analysis of inference performance with different block settings.\n", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Advantages:\n1: A novel binary architecture is proposed and achieves SOTA performance.\n\nDisadvantages:\n1. The novelty is weak. The author combines the channel projection, spatial projection, and the shortcut into a new module. The core techniques channel projection, spatial projection, and the shortcut are existing techniques. The author modified them to be suitable for the binary vision MLP.\n2. The improved performance is limited. Too many techniques (short-cut, BN+RPReLU, different MBB block settings a) are combined into one model, which only achieves 1.3% increase (71.4%->72.7%). \n3. The comparison experiment is not sufficient. Most of the compared methods are before 2020. The author should compare the proposed methods with some SOTA methods published in 2021 and 2022.\n4. The proposed method is only validated on one dataset.\n5. The paper is not well organized, and some important definitions are not given. For example, there are two pages for the preliminaries and Vision MLP Models before the proposed method. However, there are only two pages for the proposed method. The definition of two important methods Bi-MLP-S and Bi-MLP-M are missing.\n6. The devised new framework is not useful. The core contribution combining special projection and channel projection doesn’t achieve a large increase (less than 0.8%) than the simple combination of special projection and channel projection (setting 1 in Table 3).\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "PqJqC5LZPj0", "reviewer": "Reviewer_BkjE", "summary": "In this paper, the authors first analyze the difficulty of binarizing MLP models due to the amount of FC layers (1x1 conv) that limit the ability for feature representation. Then, the authors propose mulit-branch binary MLP block to improve the accuracy and reduce the OPs compared to the stoa methods.", "questions": "I don't have specific questions and suggestions for the authors. I tend to accept this paper but I don't have much experience on model quantization and binary networks. I probably change my rating after the rebuttal and discussion phase.", "limitations": "The authors didn't address their limitations and potential negative social impact. ", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n+Clear motivation and writing\n\nThe authors analyze the difficulty of binarizing MLP models and propose their solutions with MBB blocks. The writing is overall clear and straightforward.\n\n+Strong experiment results\n\nThe authors conduct experiments based on MLP models and achieve better performance with less OPs.\n\nWeaknesses:\n\n-Missing related works\n\nFor the down-sampling layers proposed in this paper, some related works have been using such architecture like [1] but missing in the related work part.\n\n[1] Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition, TPAMI 2015", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "6pUfagLgEs9", "reviewer": "Reviewer_NgiC", "summary": "This paper proposes a new architecture for binarizing the vision MLP model and mitigates the gap between the FP32 model and the 1bit version. It designs a novel MBB block with a universal shortcut to strengthen the representation ability of the binary FC layer, and redesigns the downsampling layers to reduce the computational complexity while maintaining the classification performance. The experimental results on ImageNet dataset show that the performance of the proposed binary MLP model can exceed the state-of-the-art binary CNN model, and the ablation study demonstrates the effectiveness of each proposed module.", "questions": "See weaknesses above. I suggest the author gives an example to prove that the assumption mentioned above is true in reality. Otherwise, it may influence the conclusion of the analysis.\n\nIs 1x1 conv in the proposed downsampling block in Figure 3 binarized or remain FP32?\n\nThe font in Figure 2 should be unified.\n", "limitations": "Yes.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The analysis of computational complexity and representation ability of binary conv layer and binary FC layer is inspiring. The author points out that the representation ability of a binary conv/FC layer is related to the kernel size and the input channel, which helps the design of the following MBB block. I think this analysis is also helpful for designing an effective binary CNN module.\n- The use of multi-branch block is simple yet reasonable and effective. The authors uniformly distribute computation power on different dimensions, and substantially increase the representation ability while maintaining the computational complexity of binary FC layers, compared to binary conv layer with large kernel size.\n- The experimental results show the effectiveness of the proposed architecture, and it outperforms the SOTA binary CNN model ReActNet. \n\nWeaknesses\n- This paper should conduct an ablation study to show the overall benefit of modifying the architecture. For example, the accuracy gap between FP32 and the binary version of the original MLP and the modified MLP should be reported.\n- When analyzing the representation ability and computational complexity, this paper assumes that the number of input channel and output channel of binary conv in CNN and binary FC in MLP are the same, is it true in reality?\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "-oEEWQ_UcZ5", "reviewer": "Reviewer_qm2m", "summary": "This paper presents a binary MLP model. Specifically, it designs the binary block with multiple branches (MBB blocks) and uses the shortcut to encourage the information flow. For MBB blocks, the authors proposed to  form two different multi-branch blocks by using spatial projection and channel projection simultaneously. ", "questions": "1. Any intuition about using the multi-branch blocks and the down-sampling block with multiple max-pooling branches?\n2. What is the original size of the MLP network\n3. Can the authors compare the FLOPs as well as the model size? Because MLP networks are computational efficient, but the #parameters might be much greater than CNNs.\n", "limitations": "The authors claimed the FC layers can be viewed as 1x1 Conv and thus is limited in representation ability. However, the number of channels in FC layers are usually much larger than 1x1 Convolutions used in the CNNs. In that case this claim doesn't quite hold.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "(+) The paper is well-written. The result shows it achieves even better accuracy than the state-of-the-art binary CNNs, which is impressive.\n\n(-) I didn't see a strong motivation for using multi-branch blocks.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept