Contrastive Mutual Information Maximization for Binary Neural Networks
Authors: Yuzhang Shang
Dan Xu
Ziliang Zong
Liqiang Nie
Yan Yan
Venue: ICLR 2022 Submitted (2022)
Keywords: network compression
network binarization
contrastive learning

Abstract: Neural network binarization accelerates deep models by quantizing their weights and activations into 1-bit. However, there is still a huge performance gap between Binary Neural Networks (BNNs) and their full-precision counterparts. As the quantization error caused by weights binarization has been reduced in earlier works, the activations binarization becomes the major obstacle for further improvement of the accuracy. In spite of studies about the full-precision networks highlighting the distributions of activations, few works study the distribution of the binary activations in BNNs. In this paper, we introduce mutual information as the metric to measure the information shared by the binary and the latent full-precision activations. Then we maximize the mutual information by establishing a contrastive learning framework while training BNNs. Specifically, the representation ability of the BNNs is greatly strengthened via pulling the positive pairs with binary and full-precision activations from the same input samples, as well as pushing negative pairs from different samples (the number of negative pairs can be exponentially large). This benefits the downstream tasks, not only classification but also segmentation and depth estimation, etc. The experimental results show that our method can be implemented as a pile-up module on existing state-of-the-art binarization methods and can remarkably improve the performance over them on CIFAR-10/100 and ImageNet, in addition to the good generalization ability on NYUD-v2.

Under review as a conference paper at ICLR 2022
CONTRASTIVE MUTUAL INFORMATION MAXIMIZA- TION FOR BINARY NEURAL NETWORKS
Anonymous authors Paper under double-blind review
ABSTRACT
Neural network binarization accelerates deep models by quantizing their weights and activations into 1-bit. However, there is still a huge performance gap between Binary Neural Networks (BNNs) and their full-precision counterparts. As the quantization error caused by weights binarization has been reduced in earlier works, the activations binarization becomes the major obstacle for further improvement the accuracy. In spite of studies about the full oull-precision networks highlighting the distributions of activations, few works study the distribution of the binary activations in BNNs. In this paper, we introduce mutual information as the metric measure the information shared by the binary and the latent full-pree full-precision activations. Then we maximize the mutual information by establishing a contrastive learning framework while training BNNs. Specifically, the representation ability of the BNNs is greatly strengthened via pulling the positive pairs with binary and full- precision activations from the same input samples, as well as pushing negative pairs from different samples (the number of negative pairs can be exponentially large). This benefits the downstream tasks, not only classification but also segmentation and depth estimation, etc. The experimental results show that our method can be implemented as pile-up module on existing state-of-the-ar art binarization methods and can remarkably improve the performance over them on CIFAR-10/100 and ImageNet, addition to the good generalization ability on NYUD-v2.
INTRODUCTION
Although deep neural networks (DNNs) [1] have achieved remarkable success in various com- puter vision tasks such as image classification [2] and semantic image segmentation [3], their over-paramettization problem makes them computationally expensive and storage excessive. To advance the development of deep learning towards resource-constrained edge devices, researchers proposed several neural network compression paradigms, such as network pruning [4, 5], knowledge distillation [6] and network quantization [7, 8]. Among the network quantization methods, the network binarization method [7] stands out for quantizing weights and activations (i.e. intermediate feature maps) to 11, compressing the full-precision counterpart 32x. and replacing time-consuming inner-product in full full-precision networks with efficient anor-bitcoum operation in the BNNs.
However, severe accuracy drops always exist between full-precision models and their binary counter- parts. To tackle this problem, previous works mainly focus on reducing the quantization error induced by weights binarization [9, 10], and elaborately approximating binarization function relieve the gradient mismatch in the backward propagation [11, 8]. Indeed, they achieve the state-of state-of-the-an -htthhr-rt art performance. Yet with those two paradigms developing, narrowing down the quantization error and enhancing the gradient transmission reach their bottlenecks [12, 13], since the IW32 (only quantizing the weights into 1-bit, remaining the activations 32-bit) models are capable of performing as well as the full-precision models, implying that the activations binarization becomes the main issue for further performance improvement.
To address the accuracy degradation caused by the activations binarization, a few studies propose to regulate the distributions of the binary activations, e.g. researchers in [14] design a distribution loss to explicitly regularize the activation flow; researchers in [13] propose shift the thresholds of binary activation functions to make the distribution of binary activation unbalanced. They heuristically design low-level patterns to analyze the distributions of binary activations such minimum of the

---

Under review as a conference paper at ICLR 2022
Fastares
A
Ieviac
X,
18
Z
X
BNN
Comparison
X
Comparison
Batchlorm
2
A
X
18
Z=
bn 3x3 Conv
Sinery Actinstonns
(o) Contrastive instance learning
training DMN
Figure 1: (a): In contrastive instance learning, the features from different transformations of the input image are compared to each other. (b): However BNN can yield binary activations Ae and full-preciae precision activations Ar (i.e. two transformations of an image both from the same BNN) in the same forward pass, thus the BNN can act as two image transformations in the literature of contrastive learning.
activations and the balanced property of distributions. Nevertheless, they neglect the high-level indicators of the distribution and the unique characteristics of BNN, where the binary activations and latent full-precision activations exist in the same forward pass. Thus, we argue that the high-level properties of distributions, such as correlations and dependencies between binary and full tull-precision activations should be captured and utilized.
In this work, we explore introducing mutual information for BNNs, in which the mutual information acts as fundamental quantity to measure the information amount shared by the binary and latent real-valued activations BNNs. In contrast to the aforementioned works focusing on learning the distribution of binary activations, mutual information naturally captures non-linear statistical dependencies between variables, and thus can be used as a measure of true dependence [15]. Based on this metric, propose novel method, termed Contrastive Mutual Information Maximization for Binary Neural Networks (CMIM-BNN) strategy using contrastive estimation for the mutual information maximization. As illustrated in Figure 1, we replace the data transformation module in contrastive learning with the exclusive structure in BNNs, where full-precision and binary activations are in the same forward pass. this way, contrastive learning contributes to inter-class decorrelation of binary activations, and avoids collapse solutions. In other words, our method is built upon a contrastive learning framework to learn representative binary activations, in which we pull the binary activation closer to the full-precision activation and push the binary activation further away from other binary activations in the contrastive space. Moreover, by utilizing additional MLP module to extract representations of activations, our method can explicitly capture higher-order dependencies the contrastive space. the best of our knowledge, it is the first work aiming at maximizing the mutual information of the activations in BNNs within a contrastive learning framework.
Specifically, we design a highly effective optimization
Overall, the contributions of this paper are three-fold:
Considering the distributions of activations, we propose novel CMIM framework to optimize BNNs, by maximizing the mutual information between the binary activation and its latent real- valued counterpart;
We develop an effective contrastive learning strategy to achieve the goal of mutual information maximization for BNNs, and benefited from it, the representation ability of BNNs is clearly strengthened for not only the classification task but also downstream CV tasks;
Experimental results show that our method can significantly improve the existing SOTA methods over the classification task on CIFAR-10/100 and ImageNet, e.g. 6.4% CIFAR-100 and 3.0% on ImageNet. Besides, we also demonstrate good generalization ability of the proposed CMIM on other challenging CV tasks such as depth estimation and semantic segmentation.
2 MUTUAL INFORMATION MAXIMIZATION FOR TRAINING BNNs
2.1 PRELIMINARIES
We first define Multi- Laylrr Perceptron (MLP) with K layers. For simplification of derivation, we discard the bias term of the network. Then the MLP f(x) can be denoted as:
s(W1_-. ,",, W
(WK WK- W (x),,),
2

---

Under review as a conference paper at ICLR 2022
where x is the input sample and W R2 - R (A: 1,..,,,.. I.... K) stands for the weight matrix connecting the (k - 1)-th and the k-th layer, with dx-1 and dy representing the sizes of the input and output of the k-th network layer, respectively. The (--) function performs element wise activation for the feature maps. Notably, for a convolution layer with the input map of m channels and the output map of n channels, and the size of the kernel w x h, it results in x n to x h parameters. We can re-arrange the parameters to a weight matrix of size n x (m x h x w), such that this convolution layer can also operate in the same way as the other fully-connected layers. Hence, is sufficient to consider networks with the fully-connected layers.
Based on those predefined notions, the sectional MLP f (x) with the front layers of the fxx) can be represented as:
w (W* o . (') And the MLP of can be seen as a special case in the function sequence (f [ E (1, ,K),, i.e. fk when ls K.
f (w).
(2)
Binary Neural Networks. Here, we revisit the general binarization method in [16, 7]. which maintains full-precision latent variables Wp p for gradient updates, and the k-th weight matrix W is binarized into =1, obatining the binary weight matrix W by binarize function (normally sgn(-)), Le. W = sgn(W>; Then the intermediate activation map (full-precision) of the k-th layer is produced by A W A k-1 then the same quantization method is used to binarize the full-precision activation map A (g)),, and whole forward pass of binarization is performed by iterating this process for I. times.
Mutual Information. For two discrete variables X and Y, their mutual information can be defined as [17]:
Pxy (2,
- Px(x)
(X,Y)
(3)
where Pxx (I, 9) is the joint distribution, Px(x) = Ey Pxx (2, and Px = E. (x)((
are the marginals of X and Y, respectively.
Mutual information quantifies the amount of information obtained about one random variable by observing the other random variable. is a dimensionless quantity with (generally) units of bits, and can be thought the reduction in uncertainty about one random variable given knowledge of another. High mutual information indicates a large reduction in uncertainty; low mutual information indicates small reduction; and zero mutual information between two random variables means the variables are independent. In the content of binarization, considering the binary and full-precision activations as random variables, binary activations are proceeded from their corresponding full-precision activations. Theoretically, the mutual information between those two variables should be maximized.
would like them share much information as possible, since the
2.2 CONTRASTIVE MUTUAL INFORMATION MAXIMIZATION
In the following section, we formalize the idea of constructing a Noise-Contrastin Estimation (NCE) loss maximize the mutual information between the binary and the full-precision activations. Particularly, we derive a novel CMIM loss for training BNNs, where NCE is introduced to avoid the direct mutual information computation by estimating with its lower bound in Eq. 7.
For binary network fa and its latent full-precision counterpart fF in the same training iteration, the series of their activations { } and {a (A: E {1,... KK), where A = (a A =aa k.1 .... N can be considered (ag.at) at. should share more information, iee. the mutual information of the same layer's output activations T(af,, (kk {1..... ..) should be maximized to enforce them mutually dependent.
ag A.N and series of variables. The corresponding variables
To this end, we introduce the contrastive learning framework into our targeted binarization task. The basic idea of contrastive learning is compare different views of the data (usually under different data augmentations) to calculate similarity scores [18, 19, 20, 21, 22]. This framework is suitable for case, since the binary and full-precision activations can be seen as two different views.
3

---

Under review as a conference paper at ICLR 2022
Hull Adtivation
Fal-aradiinn Activtion
MLP
MLP
BNN
BNN
MLP
MLP
Breary Activation
Attiantion
Figure 2: Feeding two images into BNN, and obtaining the three pairs of binary and full-precision activations. Our goal is to embed the activations into contrastive space, then learn from the pair correlation with the contrastive learning task Eq. 9.
For a training batch with N samples, the samples can be denoted as: {i. E ........,... (J). We feed batch of samples to the BNN and obtain K N pairs of activations (a =² which augments the data for the auxiliary task. We define a pair containing two activations from the same sample
as positive pair, Lee if = j. (a aa²) and vice versa. With the Bayes' theorem, the posterior probability of two activations from the positive pair can be formalized as:
P(ag k.s kd g a
-i=1)
P(i=j ag b.))
(4)
R. RJ P(ag
kJ k.d ar | 1i P(a
af
And the probability of activations from negative pair is: P(i *
a a
1 P(i =
k.i R.)). To simplify the NCE derivative, several works [23, 24, 25] build assumption about the a
dependence of the variables, we also use this assumption that the activations from positive pairs are dependent and the ones from negative pairs are independent, i.e. P(ag k.i ap |i= = P(a R.1 .ap)) k.) P(a kid
and P(a R. kJ (i
(.)). P) Hence, the above equation can be simplified as:
B
P(a B
P(i a Je
k:d)
(5)
k. A. P(a aF
Plap P( (N
Performing logarithm to Eq. 5 and arranging the terms, we can achieve
P(a k.b P(a))) k. K.) - (N-
P(a aF
ki k.)
log Pii = a hid aa kJ) log + (N
S log
log(NN
1).
P(a aaF
P(a k.t k.)
((aa")P((t")
P(a
(6)
Taking expectation on both sides with respect to P(a B information Eq. 3, we can derive the form of mutual information as:
) and combining the definition of mutual
largeted mutual information
P(a ar k. P(a P(a k:j
k.j
T(a a
EΣ
P(a R.d Aod log
ΣΣ Σ P(a a |i= log Pli = a a kkd) + log(N - -1)] (7)
optimized bound
E P(
hi R.j a Ao (=)) log P(i = a a log(N - 1),
where I [(a a ) is the mutual information between the binary and full- celliprecision distributions, our targeted object. Instead of directly maximizing the mutual information, we choose to maximize its lower bound the Eq. However, the distribution P(i = j| a a is hard to estimate. We take advantage of the idea of contrastive learning, and introduce a critic function h to approximate the targeted distribution [18, 19, 20]. In practice, we use the following:
exp(r(a k. k
M( 1.) = E, (xpp(r((a ) Tap
(8)
kj
4

---

Under review as a conference paper at ICLR 2022
Algorithm 1 Forward and Backward Propagation of CMIM
Require: minibatch of data samples (X, Y), current binary weight W latent full-
end for
8: For h = .,..... K, pair {
a and a negative and positive pairs, then use Eq. layer by layer to compute the NCE loss LiCE between A and A for contrastive learning;
precision weights W and learning rate ry.
Ensure: Update weights W
1: Forward Propagation:
2: for k= K do
3: Binarize latent weights: W
Combine series of NCE loss {Civor } with the classification loss L into the CMIM loss LCMIM, with Eq. 11;
sg)(W)))
4: Perform binary operation with the activations of next layer: ProduotProductt W A b):
A
10:
Backward Propagation: compute the gra- 0c
dient of the overall loss function, i.e. wwn' using the straight through estimator to tackle the sign function;
5: Perform Batch Normalization: A
BatchNorm
6: Binarize full- full-precision activations and ob-
II: Parameter Update:
update the full- Bc precision weights: W + W 1) BWw
tain binary activations: A ())):
where r temperature parameter that controls the concentration level of the distribution [6]. is important for supervised feature learning, and also necessary for tuning the concentration of a and aF on our contrastive space.
R.
k.j
Loss Function. We define the contrastive loss function CN between the k-th layer's activations
A k 3 and A as:
E E hi (=
log h(a k.
a + NE p(a ((())))))))))))))))))))))) loo((1 h(a k. ( )
(9)
We would like to comment on the above loss function from the perspective of contrastive learning. The first term of positive pairs is optimized for capturing more intra-class correlations and the second term of negative pairs is for inter-class decorrelation. Because the pair construction is instance-wise. the number of negative samples theoretically can be the size of the entire training set, e.g. 1.2 million for ImageNet. With those additional hand-craft designed contrastive pairs for the proxy optimization problem in Eq. 9, the representation capacity of BNNs can be further improved, as many contrastive learning methods demonstrated [22, 18, 19, 20].
Moreover, the optimal h = arg max CiCE can approximate the targeted distribution, i.e.
h(a a
k. = P(i ag a
(10)
where the detailed proof is shown in the supplementary material. Thus, with Eq. 7-10 derive that minimizing the NCE loss CiV is equivalent to maximizing the targeted mutual information between the binary and full-precision activations, / (a af).
Combining the series of NCE loss from different layers {Chce) (b=11 1, 1,..,, K), the overall loss f. can be defined as:
CNCE 3K-L-K
L
Latss
(11)
where Cats is the classification loss respect to the ground truth, À is used to control the degree of NCE loss, 3 is a coefficient greater than 1, and we denote the CMIM loss as LCMTM = Ek Siice LiCR Hence, the 3K-1- decreases with & increasing and consequently the BR-1-E increases. In this way, the activations latter layer can be substantially retained, which leads to better performance in practice. The complete training process of CMIM is presented in Algorithm 1.
Ji I-E
Discussion on the CMIM Loss. Besides the theoretical formulation from the perspective of mutual information maximization, we also provide an intuitive explanation about CMIM. As illustrated in Figure 2, we strengthen the representation ability of binary activations (Figure 3) via designing a proxy task with the contrastive learning framework. By embedding the activations to the contrastive
5

---

Under review as a conference paper at ICLR 2022
X
(a) XNOR [9]
(b) IR-Net [8]
(c) RBNN [10] (d) CMIM- BNN (ours)
Figure 3: t-S [26] visualization of the activations representing for random 10 classes in CIFAR LIFAR-100 100. Every color represents a different class, We can clearly witness the improvement of our method for learning better binary representations.
space and pull-and-push the paired embeddings, the BNNs learn better representations from this difficult yet effective auxiliary contrastive learning task. Note that even though we only pick up two images to formulate Figure 2. the actual number negative samples can be huge in practice (e.g. 16,384 for training ResNet-16 18 on ImageNet), benefit from the MemoryBank [24] technique.
With this property, we speculate that the contrastive pairing works as the data augmentation, which contributes to our method. This additional pairing provides more information for training the BNNs, thus our CMIM loss can be treated as an overfitting mitigated in the Section 3.2 and 3.3 to validate our speculation.
werfitting-mitigteed module. We also conduct experiments
Difference with other contrastive learning methods. The key idea of contrastive learning is to pull representations close in positive pairs and push representations apart in negative pairs in contrastive space. Several self-supervised learning methods are rooted well-established idea the mutual information maximization, such as Deep InfoMax [19], Contrastive Predictive Coding [18], MemoryBan Bank [24], Augmented Multiscale DIM [20], MoCo [21] and SimSaim [22]. These are based on NCE [23] and InfoNCE [19] which can be
lower bound mutual information [27].
The formulation of our CMIM-BNN is similar to the classic contrastive learning methods, where all inspired by NCE. However, our approach has several differences from those methods. Firstly, the training process of BNNs is different from regular network training, where binary and latent full-precision activations exist the same forward pass. We seamlessly integrate this mixed- activation property with NCE, and thus the targeted lower bound formulated for optimization is different. Secondly, the binary and full-precision weights both optimized by the NCE loss (i.e. the two view augmentation networks are optimized simultaneously in contrastive learning), yet most aforementioned contrastive learning methods optimize their view augmentation networks separately.
3 EXPERIMENTS
In this section, we first conduct experiments to compare with existing state-of-the-arr methods in image classification. Following popular settings in most studies, we use CIFAR-10/100 [28] and ImageNet ILSVRC-2012 [29] to validate the effectiveness of our proposed binarization method. Besides comparing our method with the SOTA methods, we design experiments in semantic segmentation and depth estimation tasks on the NYUD-v2 [30] dataset testify the generalization ability of our method. Meanwhile, we design series of ablation studies to verify the effectiveness of our proposed technique, and we empirically explain the efficacy of CMIM from the perspective of mitigating overfitting. All experiments implemented using Py PyTorc [31] with one NVIDIA RTX 6000 GPU while training on CIFAR- 10/ 100 and NYUD-v2. and two GPUs on ImageNet.
the BNNs are trained by CMIM for 400 epochs with batch size of 256, initial learning rate of 0.1 and cosine learning rate scheduler. We adopt SGD optimizer with momentum of 0.9 and weight decay of 1e-4. On ImageNet, binary models are trained for 100 epochs with batch size of 256. SGD optimizer is applied with momentum 0.9, weight decay of le-4, initial learning rate of 0.1 with cosine learning rate scheduler.
Experimental Setup. On CIFAR-

---

Under review as a conference paper at ICLR 2022
Table 1: Top-1 accuracy (%) CIFAR-I0 (C-10) and Table 2: Top-1 I and Top-5 accuracy on ImageNet. i rep- CIFAR-100 (C-100) test set. The higher the better. resents the architecture which varies from the standard ResNet architecture but in the same FLOPs level.
W/A denotes the bit number of veights/activations..
Topology
Method
Bit-width (WIA)
Acc.(S) (C-10)
Acc.(S) (C-100)
Topology
Method
BW (W/A)
Top-1 (%)
Top-5
(9)
Full- precision
32/32
92.1
70.7
Full-precision
32/32
DeReFa [32]
1/1
79.3
ABC Net Ne7 [37]
1/1
42.7
67.6
QSQ [33]
1/1
84.1
XNOR-Net [9]
1/1
SLB [34]
1/1
85.5
BNN+ [7]
1/1
53.0
72.6
LNS [35]
1/1
DoReFa [32]
ResNet
IR-Net [8]
1/1
86.5
65.6
ResNet-18
XNOR+ [38]
1/1
57.1
79.9
RBNN [10]
1/1
66.0
BiRcal [1]]
1/1
56.4
79.5
IR-Net+ CMIM
1/1
87.3
68.1
IR-Net [8]
I/1
58.1
80.0
RBNN CMIM
1/1
87.6
68.2
RBNN [10]
1/1
59.9
81.0
Full-precision
32/32
93.0
72.3
BiReal + CMIM
1/1
60.1
81.3
RAD [141
1/1
IR-Net + CMIM
1/1
61.2
83.0
Proxy-BNN [36]
1/1
91.8
67.2
RBNN CMIM
1/1
62.5
84.2
ResNet
IR-Net [8]
1/1
91.6
64.5
BATS 3A]S]39]
171
60.4
83.0
-18
RBNN [10]
1/1
92.2
65.3
BATS + CMIM
1/1
IR-Net CMIM
1/1
92.2
71.2
ReActNet [40]]
1/1
69.4
85,5
RBNN CMIM
121
92.8
71.4
ReActNet + CMIM
1/1
71.0
86.3
Frel-precttion
32732
94.1
73.0
XNOR [9]
1/1
90.5
Pull-precision
32/32
73.3
91.3
DoReFa [32]
1/1
90.2
ABC-Net [37]
1/1
52.4
76.5
RAD [14]
1/1
90.5
XNOR-Net [9]
1/1
53.1
76.2
VGG small
QSQ [33]
1/1
90.0
BiReal []1]
1/1
62.2
83.9
SLB [34]
1/1
92.0
ReaNet-34
XNOR++ [38]
1/1
57.1
79.9
Proxy
Troxy-BNN BNN [36]
1/1
91.8
67.2
IR-Net [8]
1/1
62.9
84.1
IR Net [8]
1/1
90.4
67.0
LNS [35]
1/1
59.4
81.7
RBNN 130]
1/1
91.3
67.4
RENN 130]
1/1
63.1
84.4
IR-Net+ CMIM
1/1
92.0
70.0
IR Net + CMIM
1/1
64.9
85.8
RBNN CMIM
1/1
92.2
71.0
RENN CMIM
1/1
65,0
85.7
3.1 EXPERIMENTAL RESULTS
CIFAR-10/100 are widely-used image classification datasets, where each consists of 50K training images and 10K testing images of size 32x32 divided into 10/100 classes. 10K training images are randomly sampled for cross-validation and the rest images are utilized for training. Data augmentation strategy includes random crop and random flipping as in [41] during training.
For ResNet-20, we compare with DoReFa [32], QSQ [33], SLB [34], LNS [35], IR-Net [8] and RBNN [10]. For ResNet- 18, RAD [14], Proxy-BNN [36], IR-Net and RBNN are chosen to be the benchmarks. For VGG-small, our method is compared with IR-Net and RBNN, etc.
As presented in Table 1, CMIM constantly outperforms other SOTA methods. On CIFAR-100, our method achieves 2.5% 6.1% and 4.0% performance improvement with ResNet-20, ResNet- and VGG-small architectures, respectively. To show the pile-up property. we add CMIM on different baseline methods, and we can obviously observe the accuracy gain with CMIM.
ImageNet is dataset with 1.2 million training images and 50k validation images equally divided into 1K classes. ImageNet has greater diversity, and its image size 469x387 (average). We report the single-crop evaluation result using 224x224 center crop from images.
For ResNet-18, we compare our method with XNOR-Net [9], ABC-Net [37], DoReFa [32], BiReal [11], XNOR+ [38], IR-Net [8], RBNN [10]. For ResNet-34, we compare our method with BiReal, IR-Net and RBNN, etc.. All experimental results are either taken from their published papers or reproduced ourselves using their code. As demonstrated in Table 2, our proposed method exceeds all the methods in both top-1 and top-5 accuracy. Particularly, CMIM achieves around 2.5% T accuracy gain with ResNet-18 18 architecture, well 1.9% Top-1 accuracy improvement with ResNet-34 architecture, compared with the state-of-the-art method RBNN.
Apart from the aforementioned methods binarizing networks with the ResNet architecture, also conduct experiments to compare with methods designing BNNs with variant architectures. For example, BATS [39] utilizes neural architecture search [42] to automatically design BNN architecture and ReActNet [40] designs BNNs with mobile-net [43] architectures. With adding CMIM module on those architectures, we observe that BNNs trained with CMIM have noticeable performance gain, which further consolidates the effectiveness of our method.
3.2 NUMBER OF NEGATIVE SAMPLES IN CMIM
The number of negative samples n nee is an important pprammerr in our method, which ensures the estimation accuracy level of the optimized distribution in Eq. 7. We perform experiments with
7

---

Under review as a conference paper at ICLR 2022
Full network
(b) Net
(C) RDNK [10]
CMIM (ours)
Effect of number of negutive CIFAR-10
( Effect of number of negative samples
CIFAR-IDO
Traning
Reshet-
Fine- mee-wwning NYUD-22
Figure 4: In-depth analysis different aspects of the proposed approach including comparison of learned correlation maps from different methods (a-d), effect number negative samples in contrastive mutual information maximization (c-f), training and testing curves (g), and comparison of fine-tuning results (h).
ResNet18 CIFAR-100 100 for parameter analysis of n_nce, with range from 2 to 2 As the results in Figure 4f presented, the accuracy arises with increasing n.nce, which also validates our speculation in the Section 2.2 that the contrastive pairing module, serving data augmentation module in training, contributes to the performance improvement of CMIM.
3.3 MITIGATE OVERFITTING
good training objective should be reflected consistent improvement the testing performance. We investigate the relation between the training and the testing accuracy across iterations. Figure 4g shows that (1) the binary ResNet-18 can reach 100% on training set of CIFAR-100, which means its representative ability is enough for this dataset; (2) on the final stage, the testing performance of the BNN trained with CMIM loss is mush better, while the training performance is relatively lower, which a clear sign of mitigating overfitting. In addition, as the results shown in the Table we can observe the phenomenon that the accuracy gain on CIFAR- 100 is more noticeable than the gain on ImageNet. This phenomenon can also be explained from the perspective of mitigating overfitting. Since the contrastive pairing (data augmentation for the proxy contrastive learning task) plays a significant role in improving the performance BNNs, and the data for training sufficient on ImageNet than on CIFAR. The overfitting issue is not that severe ImageNet. Hence, our binarization method could be more suitable for relatively data-deficient tasks.
3.4 ABLATION STUDY
We conduct series of ablative studies of our proposed method in CIFAR-10/100 and ImageNet datasets with the ResNet1 8 architecture. By adjusting the coeflicient À in the loss function LCMTM (Eq. 11), where À equals no CMIM loss are added as our baseline. we introduce IR-Net [8] as our baseline method on all the datasets. The results are shown in Table With increasing, the performance improvements show the effectiveness of CMIM loss.
the ablative studies,
Table 3: Ablation study of CMIM. The results are pre- sented in the form of accuracy rate (%). X = 0 denotes CMIM loss added, serving our baseline.
Osbaucine)
Figure 5: Results of depth estimation and segmentation.
ImageNer IK
55.08
59.29 5999
61.33
6L17
6102
6064
91.7
8

---

Under review as a conference paper at ICLR 2022
3.5 GENERALIZATION ABILITY
To study the dependence of the binary activations from the same layer, we visualize the correlation matrix of those activations by using the shade of the color to represent the cosine similarity of two activations. Red stands for two activations are similar and blue vice versa. As shown in figures 4a-4d, CMIM captures more intra-class correlations (diagonal boxes are redder) and alleviates more inter- class correlations (non-diagonal boxes are bluer). Those intensified representative activations are constructive for fine-tuning down-stream tasks. To further assess the generalization capacity of the learned binary features, we transfer the learned binary backbone to the image segmentation and depth estimation on NYUD-v2 dataset. We follow the standard pipeline for fine-tuning. A prevalent practice is to pre-train the backbone network on ImageNet and fine-tune it for the downstream tasks. Thus, we conduct experiments with Dae heads with binary ResNet18 backbone. While fine-tuning, the learning rate is initialized to 0.001 and scaled down by 10 times after every 50K iterations and we fix the binary backbone weights, only updating the task-specific heads layers. The results are presented in Figure 4h, X-axis is the depth estimation accuracy (-logrmse, higher is better), Y-axis is segmentation performance (mloU, higher is better) and the size of dot denotes the performance of classification (bigger is better). And the visualization samples are presented in Figure 5. We can witness that the models with backbone pre-trained by CMIM outperform other methods on both segmentation and depth estimation tasks.
RELATED WORK
In [7], the researchers initiate the studies of BNNs by using the sign function binarize weights and activations to +1. To eliminate the vanishing gradient issue caused by the sign function in the binarization, the straight-through estimator (STE) [44] is utilized for the network backpropagation. Based on this archetype, copious studies contribute to improving the performance of BNNs. For example, researchers in [9] disclose that the quantization error between the full-precision weights and the corresponding binarized weights is one of the major obstacles degrading the representation capabilities of BNNs. Reducing the quantization error thus becomes a fundamental research direction to improve the performance of BNNs. Researchers propose XNOR-Net [9] to introduce scaling factor calculated by L1 norm for both weights and activation functions to minimize the quantization error. Inspired by XNOR-Net, XNOR+ [38] further learns both spatial and channel-wise scaling factors to improves the performances. Bi-Real [11] proposes double residual connections with full-precision downsampling layers mitigate the excessive gradient vanishing issue caused by binarization. ProxyBNN [36] designs proxy matrix as a basis of the latent parameter space to guide the alignment of the weights with different bits by recovering the smoothness of BNNs. Those methods have advanced the network binarization techniques.
Nevertheless, we argue that those methods focusing on narrowing down the quantization error and enhancing the gradient transmission reach their bottleneck (e.g. 1W32A ResNet-18 trained by ProxyBNN achieves 67.7% Top-1 accuracy on ImageNet, while full-precision version is only 68.5%). Because they neglect the activations in BNNs, especially the relationship between the binary and latent full full-precision activations. We treat them as discrete variables and investigate them under the metric of mutual information. By maximizing the mutual information, the performance of BNNs is further improved.
5 CONCLUSION
In this paper, we investigate the activations of BNNs by introducing mutual information to mea- sure the distributional similarity between the binary and latent full-precision activations. We take advantage of the exclusive structure of the BNN, where the binary and real-valued networks exist on the same forward pass, and establish proxy contrastive learning task maximize the targeted mutual information. We name this method CMIM-BNN, Because of the push-and-pull scheme in the contrastive learning, the BNNs derived by our method have better representation ability, ben- efiting downstream tasks, such as classification and segmentation, etc.. We conduct experiments on CIFAR- 10/100, ImageNet (for classification) and NYUD-v2 (fine-tuning for depth estimation and segmentation). The results show that CMIM outperforms several state-of-the- art binarization methods on those tasks.

---

Under review as a conference paper at ICLR 2022
REFERENCES
[I] Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature, 2015.
[2] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In Neur/PS,,000
[3] Liang-Chick Chen, George Papandreou, lasonas Kokkinos, Kevin Murphy, and Alan L Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. TPAMI, 2017.
[4] Yann LeCun, John Denker, and Sara Solla. Optimal brain damage. In NewrlPS, 1989.
[5] Song Han, Huizi Mao, and William Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. In ICI.R, 2016.
[6] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. In NeurlPS, 2014.
Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El-Yaniv, and Yoshua Bengio. Binarized neural networks. NeurlPS, 2016.
[8] Haotong Qin, Ruihao Gong, Xianglong Liu, Mingzhu Shen, Ziran Wei, Fengwei Yu, and Jingkuan Song. Forward backward information retention for accurate binary neural networks. CVPR, 2020.
[9] Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xnor-net: Imagenet classifica- tion using binary convolutional neural networks. In ECCV, 2016.
[10] Mingbao Lin, Rongrong Ji, Zihan Xu, Baochang Zhang, Yan Wang, Yongjian Wu, Feiyue Huang, and Chia- Wen Lin. Rotated binary neural network. In NewrlPS, 2020.
[11] Zechun Liu, Wenhan Luo, Baoyuan Wu, Xin Yang, Wei Liu, and Kwang- Ting Cheng. Bi-real net: Binarizing deep network towards real-network performance. IJV, 2020.
[12] Zhaowei Cai, Xiaodong He, Jian Sun, and Nuno Vasconcelos. Deep learning with low precision by half-wave gaussian quantization. In CVPR, 2017.
[13] Hyungjun Kim, Jihoon Park, Changhun Lee, and Jac-Joon Kim. Improving accuracy of binary neural networks using unbalanced activation distribution. In CVPR, 2021.
[14] Ruizhou Ding. Ting-Wir Wu Chin, Zeye Liu, and Diana Marculescu. Regularizing activation distribution for training binarized deep networks. In CVPR, 2019.
[15] Shuyang Gao, Greg Ver Steeg, Aram Galstyan. Efficient estimation mutual information for strongly dependent variahles. In AISTATS, 2015.
[16] Matthieu Courbariaux, Yoshua Bengio, and Jean-Pierre David. Binaryconnect: Training deep neural networks with binary weights during propagations. In NeurlPS, 2016.
[17] Leon Brillouin. Science and information theory. Courier Corporation, 2013.
[18] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding, arXiv preprint arXiv: 1807.0378, 2018.
[19] R Devon Hjelm, Alex Fedorov, Samuel Lavoie- Marchildon, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. arXiv preprint 1808 06666700080 2018
[20] Philip Bachman, R Devon Hjelm, and William Buchwalter. Learning representations by maximizing mutual information across views. In NewrlPS, 2019.
[21] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020.
[22] Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In CVPR, 2021.
[23] Michael Gutmann and Aapo Hyvärinen. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. AISAATS, 2010.
[24] Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non-parametric instance discrimination. In CVPR, 2018.
10

---

Under review as a conference paper at ICLR 2022
[25] Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive representation distillation. In ICLR, 2021.
[26] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. JMLR, 2008
[27] Ben Poole, Sherjil Ozair, Aaron Van Den Oord, Alex Alemi, and George Tucker. On variational bounds of mutual information. In ICML, 2019.
[28] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[29] Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei. Imagenet: large-scale hierarchical image database. In CVPR, 2009.
[30] Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus. Indoor segmentation and support inference from rgbd images. In ECCV, 2012.
[31] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. In NeurIPS, 2019.
[32] Shuchang Zhou, Yuxin Wu, Zekun Ni, Xinyu Zhou, He Wen, and Yuheng Zou. Dorefa-net: Training low bitwidth convolutional neural networks with low bitwidth gradients. arXiv preprint arXiv: 060.666666666660 60, 2016.
[33] Ruihao Gong, Xianglong Liu, Shenghu Jiang. Tianxiang Li, Peng Hu, Jiazhen Lin, Fengwei Yu, and Junjic Yan. Differentiable soft quantization: Bridging full-precision and low-bit neural networks. In ICCV, 2019.
[34] Zhaohui Yang, Yunhe Wang, Kai Han, Chunjing Xu, Chao Xu, Dacheng Tao, and Chang Xu. Searching for low-hit weights in quantized neural networks. In NeurlPS, 2020.
[35] Kai Han, Yunhe Wang, Yixing Xu, Chunjing Xu, Enhua Wu, and Chang Xu. Training binary neural networks through learning with noisy supervision. In ICML, 2020.
[36] Xiangyu He, Zitao Mo, Ke Cheng, Weixiang Xu, Qinghao Hu, Peisong Wang. Qingshan Liu, and Jian Cheng. Proxybnn: Learning binarized neural networks via proxy matrices. In CVPR, 2020.
[37] Xiaofan Lin, Cong Zhao, and Wei Pan. Towards accurate binary convolutional neural network. In NeurlPS, 2017.
[38] Adrian Bulat and Georgios Tzimiropoulos. Xnor-net++:: Improved binary neural networks. In BMVC, 2019.
[39] Adrian Bulat, Brais Martinez, and Georgios Tzimiropoulos. Bats: Binary architecture search. In ECCV, 2020.
[40] Zechun Liu, Zhiqiang Shen, Marios Savvides, and Kwang- Cheng. Reactnet: Towards precise binary neural network with generalized activation functions. In ECCV, 2020.
[41] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016
[42] Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. Neural architecture search: A survey. JMLR, 2019.
[43] Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andrectto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv: 04000,,04811111 2017.
[44] Yoshua Bengio, Nicholas Léonard, and Aaron Courville, Estimating or propagating gradients through stochastic neurons for conditional computation. /388:3330..3432, 2013.
11


Reviews:
[{"review_id": "wIN0VDCdvBr", "reviewer": "Reviewer_9r7Z", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "My main concern is about the novelty. A similar contrastive distillation method has already been proposed in [1]. This submission falls into the first setting shown in Figure 1(a) of [1], which compresses the model by maximizing the mutual information between the binary student network and the full-precision teacher network. The derivations of contrastive mutual information maximization in equations (4)-(7) are almost the same as (4)-(11) in [1]. The loss function (9) in this paper also resembles that (equation (18)) of [1]. The authors should clarify the connections/differences of this submission with [1]. \n\nMinors: \n- What does \\dot mean in equation (1)?\n- Figure (4) legend should be CMIM instead of MIM.\n\n[1] CONTRASTIVE REPRESENTATION DISTILLATION, ICLR 2020.", "summary_of_the_paper": "This paper proposes to use contrastive distillation to train a binary network by maximizing the mutual information between itself (student network) and the full-precision network (teacher network). Empirical results show that this new objective further improves the binarization performance on top of several recent binary networks on image classification tasks. The authors also empirically show that models trained with the proposed contrastive objective have good transfer performance.", "summary_of_the_review": "This paper is overall structured clearly and well written. However, the novelty may be limited as it can be viewed as an application of the previous contrastive distillation method to the quantization task.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "npHAh1VTowh", "reviewer": "Reviewer_LrbF", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths**\n\n* The paper has good writing and structure, thus easy to follow.\n* The paper demonstrates promising experimental results. The results show that the proposed method improves performance on different data sets, multiple models, and multiple tasks. It shows outstanding scalability and generality.\n\n**Weaknesses**\n\n* **Limited Novelty.** However, the contribution and novelty of the proposed approach are limited. The approach looks like applying the method proposed in [1] to the BNN optimization problem. The combination is very straightforward and lacks real innovation.\n* **A convincing explanation is required.** The effect of data augmentation by utilizing negative pairs seems to be more effective, but the mutual information maximization between positive pairs lacks a convincing explanation. How do we understand the essential meaning of maximizing mutual information between fp activation and their signs? So let P denote the distribution of fp activation and Q denote the distribution of their signs (so-called binary activation in the paper). The mutual information of P and Q is the entropy of Q, since Q is always part of P. The increase of the entropy of P cannot explain the increase of the mutual information. Only when the distribution [-1, 1] of Q itself is closer to 50%:50%, the entropy is the largest. In this case, the binarization is more uniform, and it can encode more information. But in this case, mutual information maximization has the same effect as maximizing entropy(Q).\nI hope the author can provide reasonable proof or at least give a more convincing explanation of why maximizing mutual information of P and Q makes sense? At present, it feels a bit rough to apply the existing contrastive learning framework to the BNN optimization problem.\n* **AC please note that** page 5 after equation 10: “where the detailed proof is shown in the supplementary material” I am very confused, where is the author’s proof? I did not find any supplementary materials. In addition, the formula derivation in this paper has many similarities with [1]. The supplementary materials in [1] provide specific derivations of the similar formula . . .\n* **Real difference with other contrastive learning methods not found.** I can't see the obvious difference from other contrastive learning methods. It seems to be the application of existing methods in the BNN training scene.\n* **Unconvincing experimental results**: Experimental Setup on ImageNet is quite confusing. Hyperparameters are pretty different from the known open source BNN methods, including the use of weight decay (the essence of BNN training is to change the sign of weight, weight decay is almost useless, so usually wd is not used), too large initial learning rate (usually 10-100 times smaller), SGD optimizer (usually adam, see AdamBNN paper for explanation), I am curious how the author can exceed the results of ReActNet by training only 100 epochs because of the latter trains at least 256 epochs. In summary, according to the above questions, I am curious whether the author considers providing code and more experimental details such as logs and models? This will be a great help.\n\n**Minor issue**\n\n* How do you implement BATS? Since the author still didn’t share the codes as they claimed. Would you like to share your implementation? It will be a great effort.\n* The ImageNet results using ResNet18 (also ResNet34) seem to be relatively weak. Strong baselines such as ReActNet with BiReal backbone are not compared (65.9% top1 accuracy, https://github.com/liuzechun/ReActNet/). Other stronger results from RealToBinary and MeliusNet are also ignored (All of them achieved a higher accuracy with aligned computation complexity and model size).\n* Related work overlooked a lot of recent efforts on binary neural network research, e.g., ReActNet, MeliusNet, AdamBNN, etc. However, some of them are compared in the experiment section.\n\n[1] Yonglong Tian, Dilip Krishnan, Phillip Isola, Contrastive Representation Distillation,  ICLR2020 \n", "summary_of_the_paper": "This paper proposes an auxiliary method for training BNN models. It follows the idea of contrastive-mutual-information-maximization, which utilizes the full-precision and the binary activation of BNN to form positive (binary and fp activation of the same sample) and negative (binary activation of different samples) pairs for contrastive training. The auxiliary contrastive loss can provide data augmentation and effectively enhance the model's generalization ability.", "summary_of_the_review": "Regarding the existing problems in the paper, I recommend rejection for now, but it may be adjusted according to the rebuttal.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "aLGH4wEx9Ww", "reviewer": "Reviewer_vKaa", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "pros.\n1. The experiment results reported in Table 1 and Table 2 seem quite strong. \n2. The idea of applying contrastive loss to BNN training seems new. It is interesting to see that full-precision latent weights can still be used in a new manner.\n\ncons.\n1. The main concern is the direct similarity between CMIM and CRD [1]. Please refer to the \"Details Of Ethics Concerns\".\n2. Since the core part of this manuscript shares the same idea with CRD, the authors may overclaim the contribution of \"a novel CMIM framework\". In light of this, I think the novelty of this paper can be limited. \n3. Though introducing latent weights into BNN training seems new, the extra training cost remains the same as the standard Knowledge Distillation (KD) framework based on FP32 teacher networks. However, I found no ablation study on it. Why CMIM should outperform CRD (if the authors argue that CMIM is indeed different from CRD)? \n4. It is well known that KD should further improve the performance of student networks. Besides, Label Refinery [2] and Real-to-Binary [3] have shown that KD+BinConv leads to $+7$% Top-1 accuracy on ImageNet with XNOR-Res50 and $+4.3$% Top-1 accuracy on CIFAR-100 with Res18.\n5. Since CMIM achieves much higher accuracy than RBNN, it is no doubt that t-SNE results will be improved. Note that the authors tend to maximize the lower bound of the mutual information. I expect some in-depth analysis on the change of mutual information during training. \n6. How to prove that $h^*(\\mathbf{a}^{k,i}_B,\\mathbf{a}^{k,j}_F)=P(i=j|\\mathbf{a}^{k,i}_B,\\mathbf{a}^{k,j}_F)$?\n7. Is there any constraint on the form of $h^*(\\mathbf{a}^{k,i}_B,\\mathbf{a}^{k,j}_F)$? Why use Eq.(8) as the critic function to approximate the target distribution? How to determine whether $h^*(\\mathbf{a}^{k,i}_B,\\mathbf{a}^{k,j}_F)$ converges to the target distribution?\n8. Strangely, the most related work CRD is not included in section \"Difference with other contrastive learning methods.\"\n\nref:\n* [1] Contrastive Representation Distillation. ICLR2020\n* [2] Label Refinery: Improving ImageNet Classification through Label Progression. arXiv2018  \n* [3] Training Binary Neural Networks with Real-to-Binary Convolutions. ICLR2020", "summary_of_the_paper": "The authors propose to make full use of the full-precision latent weights in BNN training by utilizing the popular contrastive loss between samples generated by full-precision activations and binary counterparts. They follow the derivations of \"Contrastive Representation Distillation\" to bridge the gap between mutual information maximization and the proposed loss function. The experiment results show consistent improvements over strong baseline methods on image recognition tasks.", "summary_of_the_review": "The authors may clearly discuss the differences between CMIM and existing works. The current draft makes it hard to fully evaluate the contributions of this paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "C5Keoz4umBU", "reviewer": "Reviewer_iEui", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The idea of treating binary activations and real-valued activations as two views of the same image in self-supervised learning is interesting and makes sense. The results also well support the effectiveness of the proposed method.\n\nOverall, the submission is technically sound. I have several questions as follows:\n\n1. Does inserting the MLP layers and using the proposed CMIM module slow down the training greatly? Can the authors provide the training time comparison with the baseline method?\n\n2. Have the authors visualized the binary and real-valued activation distributions inside the BNNs trained with CMIM? Does it look significantly different from other binary neural networks?\n\nMinor issues:\n\n1. The authors mentioned researchers in [13](CVPR 2021) propose to shift the thresholds of binary activation functions. However, learning to shift the thresholds in binary neural networks is first proposed in ReActNet (ECCV 2020).\n\n2. The caption in Fig. 4: “the effect of number of negative samples in contrastive mutual information maximization (c-f)” should be “(e-f)”", "summary_of_the_paper": "This paper proposed an approach to reduce the performance gap between the binary neural networks and their real-valued counterparts via maximizing the mutual information between the binary activations and the real-valued activations that exist in the binary neural network. Specifically, the authors propose to treat binary activations and real-valued activations as two views of the same image and use contrastive learning to pull these positive pairs while pushing other negative pairs generated from different images. ", "summary_of_the_review": "In summary, this is an interesting paper with clear motivation and novel techniques. I will raise my score if my concern about the training time is well addressed. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: ## Description

The paper applies ideas from contrastive representation learning to train binary neural networks. Namely, the algorithm promotes binary representations to be similar to the full-precision representations while at the same time it promotes binary representations to be dissimilar from full-precision representations corresponding to other input images. This is enforced for activations in all layers by the added contrastive loss (9).

## Decision

The main weakness of the paper pointed by reviewers were 1) overlap of the large part of derivation with the prior work [25] Tian et al. "Contrastive representation distillation", ICLR 2020; and 2) the meaning of the derivation when applied in the setting of the paper to binary and full precision weights and its soundedness. The authors proposed their arguments for 1). The reviewers board considered these arguments and did not agree (see below). Point 2) was not addressed by authors (no paper revision, justifications, proofs corresponding to the missing supplementary). It was discussed further and was found critical (see below), such that it is a clear reason for rejection regardless of 1). Overall, the idea is interesting and the method appears to be helpful experimentally, however the paper needs a major revision that would address the two points.

## Details

### Overlap with CRD

Reviewers were in a consensus on this issue, disagreeing with authors. Since the whole derivation chain of the contrastive loss already exists in the CRD work [25], it is redundant to repeat this derivation if not raising ethical concerns. Instead an original work should review or just refer to the existing derivation and only discuss the new context and e.g. change the critic function $\hat h$. 

### Meaning of the derivation 

The reviewers have questioned the soundness of the initial criterion of MI between binary and full precision activations, as it reduces to just the entropy of binary activations. In particular, it seems very different in meaning to the contrastive loss the paper optimizes in the end. Here is additional feedback from the discussion.

1. Maximizing the entropy of binary activations with respect to the data distribution makes some sense. If a single binary activation was considered, its entropy is maximized when it is in the state 1 exactly for 50% of the data. Which makes it discriminative of the input. A similar centering can be achieved by Batch Normalization put in front of the activation -- if the preactivation distribution was symmetric, then BN would achieve the max entropy for the sign of preactivation. Such network design is not uncommon.
Maximizing the entropy of the full vector of binary activations appears more difficult. However we can also understand it as the mutual information between the input image and the layer of binary activations. Thus the criterion is to retain as much information about the input as possible. This makes sense as a regularization (often neural networks are regularized by adding data reconstruction capabilities / loss), and is aligned well with goals such as re-using the features for other tasks (as in Sec .3.5) but contradicts to some other principles proposed in the literature, e.g. the information bottleneck (that the maximum information about the target rather than the input should be preserved).
Amongst methods that study the direction of maximizing the entropy in binary networks, reviewers mention IR-Net and Regularizing Activation Distribution for Training Binarized Deep Networks. The architecture with BN before activation is used in the latter work and some more recent works, e.g. BoolNet.

2. It is not clear whether optimizing the contrastive loss retains the same meaning as maximizing MI. The derivation from CRD paper used here applies several lower bounding steps. Maybe the strongest one is that the critic is chosen to be of a specific function rather than a universal approximator. However there is no obvious gap. In fact knowing that binary activations are just a sign mapping of full precision ones, should allow one to estimate $p(i=j| a^i_B, a^j_F)$ in a simple way.

3. In the estimator $h$ in (8) the authors make a mistake (applying their and CRD theory incorrectly):
$h$ should be the probability of a conditional Bernoulli variable estimating $p(i=j| a^i_B, a^j_F)$. It should not depend on $a^j_F$ for other values of $j$ than the given one. However in the denominator in (8) it does. Therefore this estimator, and as a result the specific NCE loss proposed, appear unjustified. If the critic from CRD eq. (19) is adopted, it is not clear whether it makes sense for a pair of binary and full precision descriptors (note that for $i=j$ the scalar product between the two is just $\|a_F\|_1$).  It seems that the design of a meaningful critic is a serious gap the authors should address. Observing that the initial objective, the MI criterion, was in fact independent of full-precision states (as it is the entropy of binary states), one can propose that an appropriate critique should use binary states only, such as
$$
h(a_B^i,a_B^j) = \sigma(\left<a_B^i, a_B^j\right>  + c ).
$$
When fixing $\hat h$ the result in (10) that the maximum likelihood estimator for $p(i=j | a_B^i, a_F^j)$ with a generic neural network can approximate this distribution arbitrary well becomes irrelevant.

When the paper speaks of randomness, e.g. "binary and full precision activations as random variables, considering "i=j" as a random variable, it is needed to specify the source of randomness or the distribution, i.e. to add "for a network input drawn from the data distribution" in the first case and "under i and j picked at random uniformly in the batch" in the second.

Theoretically, the paper would become more convincing, if the the entropy of binary activations was measured by independent tools from the literature after training with and without NCE loss and it was shown that indeed the method achieves an improvement in this objective, reconfirming that the principle and the derivation were sound. An ablation study on other modifications such as weight decay may be helpful to convince researchers that the main source of improvements in experiments is the new contrastive loss. Note that not all reviewers were convinced by current experimental results due to lack of descriptions / code to fully reproduce and or lack of such ablation studies.
Decision: Reject