Year: 2021

Abstract:
Neural network binarization accelerates deep models by quantizing their weights and activations into 1-bit. However, there is still a huge performance gap between Binary Neural Networks (BNNs) and their full-precision counterparts. As the quantization error caused by weights binarization has been reduced in earlier works, the activations binarization becomes the major obstacle for further improvement of the accuracy. In spite of studies about the full-precision networks highlighting the distributions of activations, few works study the distribution of the binary activations in BNNs. In this paper, we introduce mutual information as the metric to measure the information shared by the binary and the latent full-precision activations. Then we maximize the mutual information by establishing a contrastive learning framework while training BNNs. Specifically, the representation ability of the BNNs is greatly strengthened via pulling the positive pairs with binary and fullprecision activations from the same input samples, as well as pushing negative pairs from different samples (the number of negative pairs can be exponentially large). This benefits the downstream tasks, not only classification but also segmentation and depth estimation, etc. The experimental results show that our method can be implemented as a pile-up module on existing state-of-the-art binarization methods and can remarkably improve the performance over them on CIFAR-10/100 and ImageNet, in addition to the good generalization ability on NYUD-v2.

heading: 1 INTRODUCTION
text: Although deep neural networks (DNNs) [1] have achieved remarkable success in various computer vision tasks such as image classification [2] and semantic image segmentation [3], their over-parametrization problem makes them computationally expensive and storage excessive. To advance the development of deep learning towards resource-constrained edge devices, researchers proposed several neural network compression paradigms, such as network pruning [4, 5], knowledge distillation [6] and network quantization [7, 8]. Among the network quantization methods, the network binarization method [7] stands out for quantizing weights and activations (i.e. intermediate feature maps) to ±1, compressing the full-precision counterpart 32× , and replacing time-consuming inner-product in full-precision networks with efficient xnor-bitcount operation in the BNNs.
However, severe accuracy drops always exist between full-precision models and their binary counterparts. To tackle this problem, previous works mainly focus on reducing the quantization error induced by weights binarization [9, 10], and elaborately approximating binarization function to relieve the gradient mismatch in the backward propagation [11, 8]. Indeed, they achieve the state-of-the-art performance. Yet with those two paradigms developing, narrowing down the quantization error and enhancing the gradient transmission reach their bottlenecks [12, 13], since the 1W32A (only quantizing the weights into 1-bit, remaining the activations 32-bit) models are capable of performing as well as the full-precision models, implying that the activations binarization becomes the main issue for further performance improvement.
To address the accuracy degradation caused by the activations binarization, a few studies propose to regulate the distributions of the binary activations, e.g. researchers in [14] design a distribution loss to explicitly regularize the activation flow; researchers in [13] propose to shift the thresholds of binary activation functions to make the distribution of binary activation unbalanced. They heuristically design low-level patterns to analyze the distributions of binary activations such as minimum of the
activations and the balanced property of distributions. Nevertheless, they neglect the high-level indicators of the distribution and the unique characteristics of BNN, where the binary activations and latent full-precision activations exist in the same forward pass. Thus, we argue that the high-level properties of distributions, such as correlations and dependencies between binary and full-precision activations should be captured and utilized.
In this work, we explore introducing mutual information for BNNs, in which the mutual information acts as a fundamental quantity to measure the information amount shared by the binary and latent real-valued activations in BNNs. In contrast to the aforementioned works focusing on learning the distribution of binary activations, mutual information naturally captures non-linear statistical dependencies between variables, and thus can be used as a measure of true dependence [15]. Based on this metric, we propose a novel method, termed as Contrastive Mutual Information Maximization for Binary Neural Networks (CMIM-BNN). Specifically, we design a highly effective optimization strategy using contrastive estimation for the mutual information maximization. As illustrated in Figure 1, we replace the data transformation module in contrastive learning with the exclusive structure in BNNs, where full-precision and binary activations are in the same forward pass. In this way, contrastive learning contributes to inter-class decorrelation of binary activations, and avoids collapse solutions. In other words, our method is built upon a contrastive learning framework to learn representative binary activations, in which we pull the binary activation closer to the full-precision activation and push the binary activation further away from other binary activations in the contrastive space. Moreover, by utilizing an additional MLP module to extract representations of activations, our method can explicitly capture higher-order dependencies in the contrastive space. To the best of our knowledge, it is the first work aiming at maximizing the mutual information of the activations in BNNs within a contrastive learning framework.
Overall, the contributions of this paper are three-fold:
• Considering the distributions of activations, we propose a novel CMIM framework to optimize BNNs, by maximizing the mutual information between the binary activation and its latent realvalued counterpart;
• We develop an effective contrastive learning strategy to achieve the goal of mutual information maximization for BNNs, and benefited from it, the representation ability of BNNs is clearly strengthened for not only the classification task but also downstream CV tasks;
• Experimental results show that our method can significantly improve the existing SOTA methods over the classification task on CIFAR-10/100 and ImageNet, e.g. 6.4% on CIFAR-100 and 3.0% on ImageNet. Besides, we also demonstrate good generalization ability of the proposed CMIM on other challenging CV tasks such as depth estimation and semantic segmentation.

heading: 2 MUTUAL INFORMATION MAXIMIZATION FOR TRAINING BNNS

heading: 2.1 PRELIMINARIES
text: We first define a Multi-Layer Perceptron (MLP) with K layers. For simplification of derivation, we discard the bias term of the network. Then the MLP f(x) can be denoted as:
f(W1, · · · ,WK ;x) = (WK · σ ·WK−1 · · · · · σ ·W1)(x), (1)
where x is the input sample and Wk : Rdk−1 7−→ Rdk(k = 1, ...,K) stands for the weight matrix connecting the (k− 1)-th and the k-th layer, with dk−1 and dk representing the sizes of the input and output of the k-th network layer, respectively. The σ(·) function performs element-wise activation for the feature maps. Notably, for a convolution layer with the input map of m channels and the output map of n channels, and the size of the kernel w× h, it results in m× n×w× h parameters. We can re-arrange the parameters to a weight matrix of size n × (m × h × w), such that this convolution layer can also operate in the same way as the other fully-connected layers. Hence, it is sufficient to consider networks with the fully-connected layers.
Based on those predefined notions, the sectional MLP fk(x) with the front k layers of the f(x) can be represented as:
fk(W1, · · · ,Wk;x) = (Wk · σ · · ·σ ·W1)(x). (2)
And the MLP f can be seen as a special case in the function sequence {fk}(k ∈ {1, · · · ,K}), i.e. f = fK , when k = K.
Binary Neural Networks. Here, we revisit the general binarization method in [16, 7], which maintains full-precision latent variables WF for gradient updates, and the k-th weight matrix WkF is binarized into ±1, obatining the binary weight matrix WkB by a binarize function (normally sgn(·)), i.e. WkB = sgn(WkF ). Then the intermediate activation map (full-precision) of the k-th layer is produced by AkF = W k BA k−1 B , then the same quantization method is used to binarize the full-precision activation map as AkB = sgn(A k F ), and a whole forward pass of binarization is performed by iterating this process for L times.
Mutual Information. For two discrete variables X and Y, their mutual information can be defined as [17]:
I(X,Y) = ∑ x,y PXY(x, y) log PXY(x, y) PX(x)PY(y) , (3)
where PXY(x, y) is the joint distribution, PX(x) = ∑ y PXY(x, y) and PY(y) = ∑ x PXY(x, y) are the marginals of X and Y, respectively.
Mutual information quantifies the amount of information obtained about one random variable by observing the other random variable. It is a dimensionless quantity with (generally) units of bits, and can be thought as the reduction in uncertainty about one random variable given knowledge of another. High mutual information indicates a large reduction in uncertainty; low mutual information indicates a small reduction; and zero mutual information between two random variables means the variables are independent. In the content of binarization, considering the binary and full-precision activations as random variables, we would like them share as much information as possible, since the binary activations are proceeded from their corresponding full-precision activations. Theoretically, the mutual information between those two variables should be maximized.

heading: 2.2 CONTRASTIVE MUTUAL INFORMATION MAXIMIZATION
text: In the following section, we formalize the idea of constructing a Noise-Contrastive Estimation (NCE) loss to maximize the mutual information between the binary and the full-precision activations. Particularly, we derive a novel CMIM loss for training BNNs, where NCE is introduced to avoid the direct mutual information computation by estimating it with its lower bound in Eq. 7.
For binary network fB and its latent full-precision counterpart fF in the same training iteration, the series of their activations {akB} and {akF }(k ∈ {1, · · · ,K}), where AkB = (a k,1 B , · · · ,a k,N B ) and AkF = (a k,1 F , · · · ,a k,N F ) can be considered as a series of variables. The corresponding variables (akB ,a k F ) should share more information, i.e. the mutual information of the same layer’s output activations I(akB ,a k F )(k ∈ {1, · · · ,K}) should be maximized to enforce them mutually dependent.
To this end, we introduce the contrastive learning framework into our targeted binarization task. The basic idea of contrastive learning is to compare different views of the data (usually under different data augmentations) to calculate similarity scores [18, 19, 20, 21, 22]. This framework is suitable for our case, since the binary and full-precision activations can be seen as two different views.
For a training batch with N samples, the samples can be denoted as: {xi}(i ∈ {1, · · · , N}). We feed a batch of samples to the BNN and obtain KN2 pairs of activations (ak,iB ,a k,j F ), which augments the data for the auxiliary task. We define a pair containing two activations from the same sample as positive pair, i.e. if i = j, (ak,iB ,a k,j F )+ and vice versa. With the Bayes’ theorem, the posterior probability of two activations from the positive pair can be formalized as:
P (i = j | ak,iB ,a k,j F ) =
P (ak,iB ,a k,j F | i = j) 1 N
P (ak,iB ,a k,j F | i = j) 1 N + P (a k,i B ,a k,j F | i 6= j) N−1 N
. (4)
And the probability of activations from negative pair is: P (i 6= j | ak,iB ,a k,j F ) = 1 − P (i = j | ak,iB ,a k,j F ). To simplify the NCE derivative, several works [23, 24, 25] build assumption about the dependence of the variables, we also use this assumption that the activations from positive pairs are dependent and the ones from negative pairs are independent, i.e. P (ak,iB ,a k,j F | i = j) = P (a k,i B ,a k,j F ) and P (ak,iB ,a k,j F | i 6= j) = P (a k,i B )P (a k,j F ). Hence, the above equation can be simplified as:
P (i = j | ak,iB ,a k,j F ) =
P (ak,iB ,a k,j F )
P (ak,iB ,a k,j F ) + P (a k,i B )P (a k,j F )(N − 1)
. (5)
Performing logarithm to Eq. 5 and arranging the terms, we can achieve
logP (i = j | ak,iB ,a k,j F ) = − log
[ 1 + (N − 1) P (ak,iB )P (a k,j F )
P (ak,iB ,a k,j F )
] ≤ log P (ak,iB ,a k,j F )
P (ak,iB )P (a k,j F ) − log(N − 1).
(6)
Taking expectation on both sides with respect to P (ak,iB ,a k,j F ), and combining the definition of mutual information in Eq. 3, we can derive the form of mutual information as: targeted mutual information︷ ︸︸ ︷
I(akB ,a k F ) = ∑ i ∑ j P (ak,iB ,a k,j F ) log P (ak,iB ,a k,j F ) P (ak,iB )P (a k,j F )
≥ ∑ i ∑ j P (ak,iB ,a k,j F | i = j) [ logP (i = j | ak,iB ,a k,j F ) + log(N − 1) ]
= optimized lower bound︷ ︸︸ ︷ EP (ak,iB ,ak,jF |i=j) [ logP (i = j | ak,iB ,a k,j F ) ] + log(N − 1),
(7)
where I(akB ,a k F ) is the mutual information between the binary and full-precision distributions, our targeted object. Instead of directly maximizing the mutual information, we choose to maximize its lower bound in the Eq. 7. However, the distribution P (i = j | ak,iB ,a k,j F ) is hard to estimate. We take advantage of the idea of contrastive learning, and introduce a critic function h to approximate the targeted distribution [18, 19, 20]. In practice, we use the following:
h(ak,iB ,a k,j F ) = exp(τ(ak,iB ) >ak,jF )∑
j exp(τ(a k,i B ) >ak,jF )
(8)
Algorithm 1 Forward and Backward Propagation of CMIM
Require: A minibatch of data samples (X,Y), current binary weight WkB , latent fullprecision weights WkF , and learning rate η. Ensure: Update weights WkF ′.
1: Forward Propagation: 2: for k = 1 to K − 1 do 3: Binarize latent weights: WkB ←−
sgn(WkF ); 4: Perform binary operation with the
activations of next layer: AkF ←− XnorDotProduct(WkB ,A k−1 B ); 5: Perform Batch Normalization: AkF ←− BatchNorm(AkF ); 6: Binarize full-precision activations and obtain binary activations : AkB ←− sgn(AkF );
7: end for 8: For k = 1, · · · ,K, pair { ak,iB } and { ak,jB } as negative and positive pairs, then use Eq. 9 layer by layer to compute the NCE loss LkNCE between AkB and AkF for contrastive learning;
9: Combine a series of NCE loss { LkNCE } with
the classification loss L into the CMIM loss LCMIM , with Eq. 11; 10: Backward Propagation: compute the gradient of the overall loss function, i.e. ∂L∂WB , using the straight through estimator to tackle the sign function; 11: Parameter Update: update the fullprecision weights: WiF
′ ←−WkF − η ∂L∂WkB .
where τ is a temperature parameter that controls the concentration level of the distribution [6]. τ is important for supervised feature learning, and also necessary for tuning the concentration of ak,iB and ak,jF on our contrastive space.
Loss Function. We define the contrastive loss function LkNCE between the k-th layer’s activations AkB and A k F as:
LkNCE = EP (ak,iB ,ak,jF |i=j) [ log h(ak,iB ,a k,j F ) ] +NEP (ak,iB ,ak,jF |i6=j) [ log(1− h(ak,iB ,a k,j F )) ] . (9)
We would like to comment on the above loss function from the perspective of contrastive learning. The first term of positive pairs is optimized for capturing more intra-class correlations and the second term of negative pairs is for inter-class decorrelation. Because the pair construction is instance-wise, the number of negative samples theoretically can be the size of the entire training set, e.g. 1.2 million for ImageNet. With those additional hand-craft designed contrastive pairs for the proxy optimization problem in Eq. 9, the representation capacity of BNNs can be further improved, as many contrastive learning methods demonstrated [22, 18, 19, 20].
Moreover, the optimal ĥ = argmaxh LkNCE can approximate the targeted distribution, i.e.
ĥ(ak,iB ,a k,j F ) = P (i = j | a k,i B ,a k,j F ), (10)
where the detailed proof is shown in the supplementary material. Thus, with Eq. 7-10 we can derive that minimizing the NCE loss LkNCE is equivalent to maximizing the targeted mutual information between the binary and full-precision activations, I(akB ,a k F ).
Combining the series of NCE loss from different layers { LkNCE } , (k = 1, · · · ,K), the overall loss L can be defined as:
L = λ K∑ k=1 LkNCE βK−1−k + Lcls, (11)
where Lcls is the classification loss respect to the ground truth, λ is used to control the degree of NCE loss, β is a coefficient greater than 1, and we denote the CMIM loss as LCMIM = ∑K k=1 LkNCE βK−1−k . Hence, the βK−1−k decreases with k increasing and consequently the L k NCE
βK−1−k increases. In this
way, the activations of latter layer can be substantially retained, which leads to better performance in practice. The complete training process of CMIM is presented in Algorithm 1.
Discussion on the CMIM Loss. Besides the theoretical formulation from the perspective of mutual information maximization, we also provide an intuitive explanation about CMIM. As illustrated in Figure 2, we strengthen the representation ability of binary activations (Figure 3) via designing a proxy task with the contrastive learning framework. By embedding the activations to the contrastive
space and pull-and-push the paired embeddings, the BNNs can learn better representations from this difficult yet effective auxiliary contrastive learning task. Note that even though we only pick up two images to formulate Figure 2, the actual number of negative samples can be huge in practice (e.g. 16,384 for training ResNet-18 on ImageNet), benefit from the MemoryBank [24] technique.
With this property, we speculate that the contrastive pairing works as the data augmentation, which contributes to our method. This additional pairing provides more information for training the BNNs, thus our CMIM loss can be treated as an overfitting-mitigated module. We also conduct experiments in the Section 3.2 and 3.3 to validate our speculation.
Difference with other contrastive learning methods. The key idea of contrastive learning is to pull representations close in positive pairs and push representations apart in negative pairs in a contrastive space. Several self-supervised learning methods are rooted in well-established idea of the mutual information maximization, such as Deep InfoMax [19], Contrastive Predictive Coding [18], MemoryBank [24], Augmented Multiscale DIM [20], MoCo [21] and SimSaim [22]. These are based on NCE [23] and InfoNCE [19] which can be seen as a lower bound on mutual information [27].
The formulation of our CMIM-BNN is similar to the classic contrastive learning methods, where we all are inspired by NCE. However, our approach has several differences from those methods. Firstly, the training process of BNNs is different from regular network training, where binary and latent full-precision activations exist in the same forward pass. We seamlessly integrate this mixedactivation property with NCE, and thus the targeted lower bound formulated for optimization is different. Secondly, the binary and full-precision weights are both optimized by the NCE loss (i.e. the two view augmentation networks are optimized simultaneously in contrastive learning), yet most aforementioned contrastive learning methods optimize their view augmentation networks separately.

heading: 3 EXPERIMENTS
text: In this section, we first conduct experiments to compare with existing state-of-the-art methods in image classification. Following popular settings in most studies, we use CIFAR-10/100 [28] and ImageNet ILSVRC-2012 [29] to validate the effectiveness of our proposed binarization method. Besides comparing our method with the SOTA methods, we design experiments in semantic segmentation and depth estimation tasks on the NYUD-v2 [30] dataset to testify the generalization ability of our method. Meanwhile, we design a series of ablation studies to verify the effectiveness of our proposed technique, and we empirically explain the efficacy of CMIM from the perspective of mitigating overfitting. All experiments are implemented using PyTorch [31] with one NVIDIA RTX 6000 GPU while training on CIFAR-10/100 and NYUD-v2, and two GPUs on ImageNet.
Experimental Setup. On CIFAR-10/100, the BNNs are trained by CMIM for 400 epochs with batch size of 256, initial learning rate of 0.1 and cosine learning rate scheduler. We adopt SGD optimizer with momentum of 0.9 and weight decay of 1e-4. On ImageNet, binary models are trained for 100 epochs with batch size of 256. SGD optimizer is applied with momentum of 0.9, weight decay of 1e-4, initial learning rate of 0.1 with cosine learning rate scheduler.

heading: 3.1 EXPERIMENTAL RESULTS
text: CIFAR-10/100 are widely-used image classification datasets, where each consists of 50K training images and 10K testing images of size 32×32 divided into 10/100 classes. 10K training images are randomly sampled for cross-validation and the rest images are utilized for training. Data augmentation strategy includes random crop and random flipping as in [41] during training.
For ResNet-20, we compare with DoReFa [32], QSQ [33], SLB [34], LNS [35], IR-Net [8] and RBNN [10]. For ResNet-18, RAD [14], Proxy-BNN [36], IR-Net and RBNN are chosen to be the benchmarks. For VGG-small, our method is compared with IR-Net and RBNN, etc.
As presented in Table 1, CMIM constantly outperforms other SOTA methods. On CIFAR-100, our method achieves 2.5%, 6.1% and 4.0% performance improvement with ResNet-20, ResNet-18 and VGG-small architectures, respectively. To show the pile-up property , we add CMIM on different baseline methods, and we can obviously observe the accuracy gain with CMIM.
ImageNet is a dataset with 1.2 million training images and 50k validation images equally divided into 1K classes. ImageNet has greater diversity, and its image size is 469×387 (average). We report the single-crop evaluation result using 224×224 center crop from images.
For ResNet-18, we compare our method with XNOR-Net [9], ABC-Net [37], DoReFa [32], BiReal [11], XNOR++ [38], IR-Net [8], RBNN [10]. For ResNet-34, we compare our method with BiReal, IR-Net and RBNN, etc.. All experimental results are either taken from their published papers or reproduced ourselves using their code. As demonstrated in Table 2, our proposed method exceeds all the methods in both top-1 and top-5 accuracy. Particularly, CMIM achieves around 2.5% Top-1 accuracy gain with ResNet-18 architecture, as well as 1.9% Top-1 accuracy improvement with ResNet-34 architecture, compared with the state-of-the-art method RBNN.
Apart from the aforementioned methods binarizing networks with the ResNet architecture, we also conduct experiments to compare with methods designing BNNs with variant architectures. For example, BATS [39] utilizes neural architecture search [42] to automatically design a BNN architecture and ReActNet [40] designs BNNs with mobile-net [43] architectures. With adding CMIM module on those architectures, we observe that BNNs trained with CMIM have noticeable performance gain, which further consolidates the effectiveness of our method.

heading: 3.2 NUMBER OF NEGATIVE SAMPLES IN CMIM
text: The number of negative samples n nce is an important hyper-parameter in our method, which ensures the estimation accuracy level of the optimized distribution in Eq. 7. We perform experiments with
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31
Unnamed: 0 0 1 2 3 4 5 6 7 8 9
10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30
0
50
100
150
200
250
300
(a) Full network
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31
Unnamed: 0 0 1 2 3 4 5 6 7 8 9
10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30
60
40
20
0
20
40
60
80
(b) IR-Net [8]
68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99
68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99
20
0
20
40
60
(c) RBNN [10]
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31
Unnamed: 0 0 1 2 3 4 5 6 7 8 9
10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30
0
200
400
600
800
(d) CMIM (ours)
9 1 . 3 1
9 1 . 5 3
9 1 . 9 9
9 2 . 1 7 9 2 . 1 8
9 2
9 1 . 7 9 9 1 . 7 2 Ac cur
acy (%
)
N u m b e r o f n e g a t i v e s a m p l e s
R e s N e t 1 8 o n C I F A R - 1 0
(e) Effect of number of negative samples on CIFAR-10
6 9 . 7 6 6 6 7
7 0 . 1 3 3 3 3 7 0 . 3 6 6 6 7
7 0 . 5 6 6 6 7
7 0 . 8 6 6 6 7 7 1 . 0 6 6 6 7
7 1 . 2 3 3 3 37 1 . 2
Ac cur
acy (%
)
N u m b e r o f n e g a t i v e s a m p l e s
R e s N e t 1 8 o n C I F A R - 1 0 0
(f) Effect of number of negative samples on CIFAR-100
0 5 0 1 0 0 1 5 0 2 0 0 2 5 0 3 0 0 3 5 0 4 0 0 0
1 0 2 0 3 0 4 0 5 0 6 0 7 0 8 0 9 0
1 0 0
Co rre
ct Ra
te (%
)
E p o c h
T r a i n i n g _ M I M T r a i n i n g _ w o _ M I M T e s t i n g _ M I M T e s t i n g _ w o _ M I M
(g) Training & testing curves of ResNet18 w/ and w/o CMIM
0.10 0.08 0.06 0.04 0.02 0.00 0.02 0.04 -logrmse
15
16
17
18
19
20
21
22
23
24
m Io
U
IR-Net
XNOR++
BiReal
XNOR RBNN
BATS
MIM
(h) Fine-tuning results on NYUD-v2.
Figure 4: In-depth analysis on different aspects of the proposed approach including a comparison of learned correlation maps from different methods (a-d), the effect of number of negative samples in contrastive mutual information maximization (c-f), training and testing curves (g), and a comparison of fine-tuning results (h).
ResNet18 on CIFAR-100 for parameter analysis of n nce, with range from 20 to 215. As the results in Figure 4f presented, the accuracy arises with increasing n nce, which also validates our speculation in the Section 2.2 that the contrastive pairing module, serving as a data augmentation module in training, contributes to the performance improvement of CMIM.

heading: 3.3 MITIGATE OVERFITTING
text: A good training objective should be reflected in consistent improvement in the testing performance. We investigate the relation between the training and the testing accuracy across iterations. Figure 4g shows that (1) the binary ResNet-18 can reach 100% on training set of CIFAR-100, which means its representative ability is enough for this dataset; (2) on the final stage, the testing performance of the BNN trained with CMIM loss is mush better, while the training performance is relatively lower, which is a clear sign of mitigating overfitting. In addition, as the results shown in the Table 3, we can observe the phenomenon that the accuracy gain on CIFAR-100 is more noticeable than the gain on ImageNet. This phenomenon can also be explained from the perspective of mitigating overfitting. Since the contrastive pairing (data augmentation for the proxy contrastive learning task) plays a significant role in improving the performance of BNNs, and the data for training is sufficient on ImageNet than on CIFAR. The overfitting issue is not that severe on ImageNet. Hence, our binarization method could be more suitable for relatively data-deficient tasks.

heading: 3.4 ABLATION STUDY
text: We conduct a series of ablative studies of our proposed method in CIFAR-10/100 and ImageNet datasets with the ResNet18 architecture. By adjusting the coefficient λ in the loss function LCMIM (Eq. 11), where λ = 0 equals to no CMIM loss are added as our baseline. In the ablative studies, we introduce IR-Net [8] as our baseline method on all the datasets. The results are shown in Table 3. With λ increasing, the performance improvements show the effectiveness of CMIM loss.

heading: 3.5 GENERALIZATION ABILITY
text: To study the dependence of the binary activations from the same layer, we visualize the correlation matrix of those activations by using the shade of the color to represent the cosine similarity of two activations. Red stands for two activations are similar and blue vice versa. As shown in figures 4a-4d, CMIM captures more intra-class correlations (diagonal boxes are redder) and alleviates more interclass correlations (non-diagonal boxes are bluer). Those intensified representative activations are constructive for fine-tuning down-stream tasks. To further assess the generalization capacity of the learned binary features, we transfer the learned binary backbone to the image segmentation and depth estimation on NYUD-v2 dataset. We follow the standard pipeline for fine-tuning. A prevalent practice is to pre-train the backbone network on ImageNet and fine-tune it for the downstream tasks. Thus, we conduct experiments with DeepLab heads with binary ResNet18 backbone. While fine-tuning, the learning rate is initialized to 0.001 and scaled down by 10 times after every 50K iterations and we fix the binary backbone weights, only updating the task-specific heads layers. The results are presented in Figure 4h, X-axis is the depth estimation accuracy (-logrmse, higher is better), Y-axis is segmentation performance (mIoU, higher is better) and the size of dot denotes the performance of classification (bigger is better). And the visualization samples are presented in Figure 5. We can witness that the models with backbone pre-trained by CMIM outperform other methods on both segmentation and depth estimation tasks.

heading: 4 RELATED WORK
text: In [7], the researchers initiate the studies of BNNs by using the sign function to binarize weights and activations to ±1. To eliminate the vanishing gradient issue caused by the sign function in the binarization, the straight-through estimator (STE) [44] is utilized for the network backpropagation. Based on this archetype, copious studies contribute to improving the performance of BNNs. For example, researchers in [9] disclose that the quantization error between the full-precision weights and the corresponding binarized weights is one of the major obstacles degrading the representation capabilities of BNNs. Reducing the quantization error thus becomes a fundamental research direction to improve the performance of BNNs. Researchers propose XNOR-Net [9] to introduce a scaling factor calculated by L1 norm for both weights and activation functions to minimize the quantization error. Inspired by XNOR-Net, XNOR++ [38] further learns both spatial and channel-wise scaling factors to improves the performances. Bi-Real [11] proposes double residual connections with full-precision downsampling layers to mitigate the excessive gradient vanishing issue caused by binarization. ProxyBNN [36] designs a proxy matrix as a basis of the latent parameter space to guide the alignment of the weights with different bits by recovering the smoothness of BNNs. Those methods have advanced the network binarization techniques.
Nevertheless, we argue that those methods focusing on narrowing down the quantization error and enhancing the gradient transmission reach their bottleneck (e.g. 1W32A ResNet-18 trained by ProxyBNN achieves 67.7% Top-1 accuracy on ImageNet, while full-precision version is only 68.5%). Because they neglect the activations in BNNs, especially the relationship between the binary and latent full-precision activations. We treat them as discrete variables and investigate them under the metric of mutual information. By maximizing the mutual information, the performance of BNNs is further improved.

heading: 5 CONCLUSION
text: In this paper, we investigate the activations of BNNs by introducing mutual information to measure the distributional similarity between the binary and latent full-precision activations. We take advantage of the exclusive structure of the BNN, where the binary and real-valued networks exist on the same forward pass, and establish a proxy contrastive learning task to maximize the targeted mutual information. We name this method CMIM-BNN. Because of the push-and-pull scheme in the contrastive learning, the BNNs derived by our method have better representation ability, benefiting downstream tasks, such as classification and segmentation, etc.. We conduct experiments on CIFAR-10/100, ImageNet (for classification) and NYUD-v2 (fine-tuning for depth estimation and segmentation). The results show that CMIM outperforms several state-of-the-art binarization methods on those tasks.

Review summaries:
The authors propose to make full use of the full-precision latent weights in BNN training by utilizing the popular contrastive loss between samples generated by full-precision activations and binary counterparts. They follow the derivations of "Contrastive Representation Distillation" to bridge the gap between mutual information maximization and the proposed loss function. The experiment results show consistent improvements over strong baseline methods on image recognition tasks.

This paper proposes to use contrastive distillation to train a binary network by maximizing the mutual information between itself (student network) and the full-precision network (teacher network). Empirical results show that this new objective further improves the binarization performance on top of several recent binary networks on image classification tasks. The authors also empirically show that models trained with the proposed contrastive objective have good transfer performance.

This paper proposes an auxiliary method for training BNN models. It follows the idea of contrastive-mutual-information-maximization, which utilizes the full-precision and the binary activation of BNN to form positive (binary and fp activation of the same sample) and negative (binary activation of different samples) pairs for contrastive training. The auxiliary contrastive loss can provide data augmentation and effectively enhance the model's generalization ability.