Boolean Logic for Low-Energy Deep Learning

Authors: van minh nguyen
cristian  ocampo
aymen  askri
ba-hien  tran
Venue: ICML_cc_2024_Workshop_WANT
Published: 2024-05-29 11:17:40

Abstract: Deep learning is computationally intensive. Much effort has been given to reduce the arithmetic complexity whilst energy consumption is the most relevant bottleneck, in which data movement is the dominant part. In addition, the literature focus has been on inference whereas training is several times more intense. In this paper, we make use of the Boolean neuron design and Boolean logic backpropagation to train deep models in the binary domain using Boolean logic instead of gradient descent and real arithmetic. We propose a detailed energy evaluation for both training and inference phases. Our method achieves the best results in standard image classification tasks and consumes almost 27 times less energy with our most efficient and best performing Boolean network. This energy efficiency paves the way for an edge device use, in particular for fine-tuning large models on a dedicated task. In practice, our approach outperforms the state-of-the-art semantic segmentation and shows promising image super-resolution performance.

Main paper: 
 introduction: 
 Running deep models, i.e., inference, requires considerable computational resources, it is yet the tip of the iceberg. Deep model training is an iterative process involving abundant computation and data for learning. It incurs storing multiple temporal variables and buffers for gradient computation and parameter optimization. This intense process is further repeated for hyper-parameter tuning, running for weeks or months on specialized equipment, resulting in another order of magnitude of carbon footprint and computational resource requirement (Strubell et al., 2019) .The vast majority of works that target the resource con-Accepted to the Workshop on Advancing Neural Network Training at International Conference on Machine Learning (WANT@ICML 2024).strained training bottleneck focus on the number of arithmetic operations (FLOPS or BOPS) (García-Martín et al., 2019; Qin et al., 2020a) rather than the consumed energy/memory. However, it has been shown that OPs number is meaningless, even harmful, because it does not map directly to actual system complexity. Instead, energy and memory consumption are the consistent and efficient measures of computing hardware (Sze et al., 2017; 2020; Yang et al., 2017; Strubell et al., 2019) . In particular, data movement dominates computing in energy consumption and is strictly tied to system architecture, memory hierarchy, and dataflow (Kwon et al., 2019; Sim et al., 2019; Yang et al., 2020a; Chen et al., 2016) . Therefore, design effort subjected to reducing OPs alone is inefficient. Currently, the main approach to tackle such bottleneck is quantization. It is becoming popular for LLMs (Frantar et al., 2022; Lin et al., 2023; Kim et al., 2023) to enable inference on affordable devices. But only post-training quantization is available on standard GPUs. Better quantized models can be obtained through quantization-aware training (Gupta et al., 2015; Zhang et al., 2018a; Jin et al., 2021; Yamamoto, 2021; Huang et al., 2021; Umuroglu et al., 2017) , and quantized training (Chen et al., 2020; Sun et al., 2020; Yang et al., 2022; Chmiel et al., 2021) , that reduce the numeric precision of weights, activations, and dataflow from full-precision (FP) to finite-precision format.A special case of quantization-aware training is binarized neural networks (BNNs) which were first proposed by Courbariaux et al. (2015; 2016) and have been followed by a huge amount of subsequent contributions (Gholami et al., 2022; Zhao et al., 2020; Guo, 2018; Nagel et al., 2021) . This design usually binarizes weights and activations to obtain principal forward computation blocks in binary. It learns binarized weights via full-precision latent ones, which are updated by the classical gradient descent backpropagation. The gradient of the binarized variables is usually approximated by a differential proxy of the binarization function, which is most often the identity proxy. Many concurrent approaches (Bai et al., 2018; Ajanthan et al., 2019; 2021; Lin et al., 2020b; Leconte et al., 2023) formulated the BNN learning task as a constrained optimization problem and discussed different methods to generate binary weights from real-valued latent ones. In practice, these works showed that BNNs could achieve state-of-the-art accuracy in study-level classification problems such as CIFAR-10 or MNIST, but suffer significant accuracy drop on more challenging problems such as ImageNet (Rastegari et al., 2016; Qin et al., 2020a) . Besides the reduced network approximation capacity due to lower data precision (Zhou et al., 2016) , the use of full-precision optimizers for estimating binary weights are the causes of this degradation. To compensate for this accuracy loss, most recent prominent works (Liu et al., 2020; Nie et al., 2022; Guo et al., 2022) used multiple full-precision components in the network, whereas only a few dataflows have remained binary.In this work we aim to answer whether energy-friendly deep learning is possible both for inference and training all while maintaining performance. For that, we explore Boolean notions to define networks that are predominantly Boolean, with low energy demands, and that are trained in the binary domain. Extensive experimental evaluation is conducted on a set of common vision tasks requiring moderate to higher levels of accuracy.Our contributions are: (1) We make use of Boolean neuron design and Boolean logic backpropagation principle for the training of deep models in binary domain. (2) We introduce a new methodology to fairly assess the energy complexity of both training and inference phases. (3) We show competitive or state-of-the-art results of this design in complex computer vision-related tasks, including image super-resolution. (4) We show that employing pre-trained Boolean NNs for edge device fine-tuning tasks, such as classification and segmentation, yields very good performances at low energy cost. 
related works: 
 Energy consumption is a fundamental metric for measuring hardware complexity. However, it requires specific knowledge of computing systems and makes it hard to estimate. Only few results are available, though experimental-based and limited to specific tested models, e.g., Gao et al. (2020) ; Shao & Brooks (2013) ; Mei et al. (2014) ; Bianco et al. (2018) ; Canziani et al. (2016) ; García-Martín et al. (2019) . Although experimental evaluation is precise, it requires considerable implementation efforts while not generalizing. In addition, most relevant works are only limited to inference and not training (Chen et al., 2016; Kwon et al., 2019; Yang et al., 2020a) . Therefore, developing an analytic method to efficiently estimate training energy consumption is desirable.Regarding NN architectures, significant advances have been made on BNNs (Binarized Neural Networks) for the Ima-geNet classification task, driving their performance to higher grounds (Guo et al., 2022; Liu et al., 2020; Tu et al., 2022; Lee et al., 2022; Zhang et al., 2022; Xing et al., 2022; Martinez et al., 2020; Wang et al., 2023b) . These works, which attempt to reduce the accuracy gap between BNNs and fullprecision networks, typically target the primary sources of the computational burden of CNNs, essentially convolutions, data-streams memory (both numeric type and size) and network depth. Consequently, modern BNNs are improved over the following three main areas.Binarization strategy. It seeks to efficiently binarize realvalued data. The sign function is the primary alternative to binarize data-streams, with additional constraints on the data like clipping (Zhang et al., 2022; Guo et al., 2022) . Re-ActNet (Liu et al., 2020) is a prominent work that proposes RSign, a more general alternative to sign, which deals with the fact that distributions may be shifted or biased. A more recent option, Tu et al. (2022) argues that binary values {1, -1} might restrain the approximation capabilities of BNNs, which is why they binarize activations to two real values for more representative power.Optimization strategy. Since latent-based training (Courbariaux et al., 2015) remains the underlying method for updating binarized weights, a differential proxy of sign is required. Different or modified alternatives to straightthrough-estimator (STE) have been proposed (Liu et al., 2020) . Piece-wise polynomials and hyper-parameterized tanh have been used (Nie et al., 2022) . The latent-based approach requires storing both binary and real parameters during training. Furthermore, this approach typically requires a sequential training of multiple stages where activations and weights get progressively converted from full-precision to binary types (Guo et al., 2022; Zhang et al., 2022; Xing et al., 2022) , resulting in longer training times. Modern BNN methodologies (Xing et al., 2022; Liu et al., 2020; Guo et al., 2022; Zhang et al., 2022; Tu et al., 2022; Lee et al., 2022; Liu et al., 2022) agree on the fact that using knowledge distillation (KD) closes the gap between BNNs and full-precision models for which additional data augmentation is needed to reduce the common overfitting of BNNs (Guo et al., 2022) . In most cases, a single teacher like ResNet34 or ResNet50 suffices to significantly increase the accuracy. More recently, using multi-KD with four teachers, BNext (Guo et al., 2022) , reached performances not reported before. In this sense, existing works have investigated BNNs from the perspective that network binarization is considered as a plugin feature to an existing full-precision DNN. For KD, the goal is to transfer the knowledge of a teacher model to a smaller student model. If trained from scratch, the student model generally performs worse than its teacher. However, under the supervision of the teacher network, the binary network can preserve the learning capability and thus obtain comparable performance to the teacher network. Consequently, the process still requires full-precision model training, and cannot tackle the complexity problem of the network training. Furthermore, KD-based training depends on specialized teachers on a particular task, thus reducing functionality on new data. Helwegen et al. (2019) ; Wang et al. (2021) proposed some heuristic and improvement of the classic BNN latent-based optimizer.Architecture design. ResNet (Liu et al., 2020; 2018; Guo et al., 2022; Bethge et al., 2020) and MobileNet (Liu et al., 2020; Guo et al., 2022) are the most frequent layouts. In (Zhou et al., 2016; Rastegari et al., 2016) the authors experimented with Alexnet. Among these methodologies, the basic blocks have been greatly transformed. Common modifications include additional shortcuts, automatic channel scaling with Squeeze-and-Excitation (Zhang et al., 2022) or block duplication plus concatenation in the channel domain (Liu et al., 2020; Guo et al., 2022) . Recent alternatives incorporate modules that better adapt the input domain to binary dataflows (Xing et al., 2022) , replace standard convolutions with lighter pointwise convolutions (Liu et al., 2022) , or propose 1-bit alternatives of linear projections (Wang et al., 2023a) .Since the release of ReActNet, the best results are obtained by alternating low-precision dataflows to full-precision after every binary convolution within the network. These works substantially rely on real-valued dataflows during feedforward such as PReLU, Batch Normalization, FP scaling, and further boost accuracy via KD. This highlights the need for native binary neural networks (Nguyen, 2023) , and a precise complexity evaluation method to be able to assess gains in regards of memory, energy, and latency. 
boolean training: 
 The design and training of Boolean layers follows the principle proposed by Nguyen (2023) . For illustration purpose, Algorithm 1 presents a pseudo code of our implementation of a Boolean fully-connected layer that uses Boolean logic B e.g. XOR, XNOR, or any other. In the forward pass, at iteration t, input x l,t is buffered for later use in the backward, and the jth neuron output at kth sample is computed as:EQUATION∀k ∈ [1, K], ∀j ∈ [1, n]where K, m, n are, resp., the training mini-batch, layer input and output size. Here, to take into account the case of real-valued input data, definitions due to Nguyen (2023) are presented in the following. For x ∈ R, define x bool = True ⇔ x ≥ 0, and x bool = False ⇔ x < 0, and |x| its magnitude. Then, logic operation between x ∈ R and b ∈ {False, True} is defined according to (Nguyen, 2023) In the backward pass, this layer receives g l+1,t from downstream layer l + 1, which is usually an activation or a batch normalization (BN) layer. The backpropagation through binary activation function is considered for two cases: without vs. with BN. In the first case, arithmetic layer is directly followed by the binary activation to avoid BN for complexity reduction, tanh is used as the backward activation. In the second case, a simple approximation such as the one proposed by (Liu et al., 2018 ) can be used. Then, backpropagation signal g l,t , cf . line 7 in Algorithm 1, is computed following Nguyen (2023) as:g l,t k,i = n j=1 1 g l,t k,i,j = True |g l,t k,i,j | - n j=1 1 g l,t k,i,j = False |g l,t k,i,j |, (2) ∀k ∈ [1, K], ∀i ∈ [1, m],where g l,t k,i,j is given according to Nguyen (2023) for the utilized logic B, for example:g l,tk,i,j = XNOR(g l+1,t k,j , ¬w l,t i,j ), for XOR neuron, XNOR(g l+1,t k,j , w l,t i,j ), for XNOR neuron, in which ¬ is the logic negation. Optimization signal at line 10 in Alogrithm 1 is given according to (Nguyen, 2023) as:q l,t+1 i,j = K k=1 1 q l,t i,j,k = True |q l,t i,j,k | - K k=1 1 q l,t i,j,k = False |q l,t i,j,k |, (3) ∀i ∈ [1, m], ∀j ∈ [1, n] where q l,t i,j,k = XNOR(g l+1,t k,j , ¬x l,t k,i ), for XOR neuron, XNOR(g l+1,t k,j , x l,t k,i ), for XNOR neuron.Finally, the weights are updated in lines 13-18 of Algorithm 1 just following the rule formulated in Nguyen (2023).Energy consumption is a fundamental metric for measuring hardware complexity. However, it requires specific knowledge of computing systems and makes it hard to estimate. Few results are available, though experimental-based and limited to specific tested models, e.g. (Gao et al., 2020; Shao & Brooks, 2013; Mei et al., 2014; Bianco et al., 2018; Canziani et al., 2016; García-Martín et al., 2019) . Although experimental evaluation is precise, it requires considerable implementation efforts while not generalizing. In addition, most relevant works are only limited to inference and not training. (Chen et al., 2016; Kwon et al., 2019; Yang et al., 2020a) .In this work, we intend to estimate the training energy consumption on Ascend chip architecture introduced in Liao et al. ( 2021) and dedicated to DNN computing. Its evaluation on GPU hardware is subject to future work. Ascend architecture has achieved important commercial successes with more than 100 million chips that have been used in real products for applications ranging from smartwatches, smartphones, and smart cars to intelligent clouds.The core design of Ascend is well described in (Liao et al., 2021) . Essentially, it introduces a 3D (cube) computing unit, providing the bulk of high-intensity computation and increasing data reuse. On the other hand, it provides multiple levels of on-chip memory. In particular, memory L0, which is nearest to the computing cube, is tripled to boost further near-memory computing capability, namely L0-A dedicated to the left-hand-side (LHS) input data, L0-B dedicated to RHS input data, and L0-C for the output. For instance, in a convolution, L0-A, L0-B, and L0-C correspond to the input feature maps (ifmaps), filters, and output feature maps (ofmaps), resp. In addition, the output results going through L0-C can be processed by a Vector Unit for in-place operations such as normalization and activation. Table 1 shows energy efficiency and capacity of the memory hierarchy of a commercial Ascend architecture (Liao et al., 2021) .Energy consumption is the sum of compute and memory energies. Compute energy is simply given by the number of arithmetic operations multiplied by their unit cost. The number of arithmetic operations is directly determined from the layer's parameters. Their unit cost is obtained by considering the compute efficiency at 1.7 TOPS/W (Liao et al., 2021) . For Boolean logic operations, we follow the usual estimation that ADD INT-n costs (2n -1) logic operations where n stands for bitwidth.On the other hand, memory energy is all consumed for moving data between their storage through memory levels and the computing unit during the entire lifetime of the process.Since energy consumed at each memory level is given by the number of data accesses to that level times per-access energy cost, it consists in determining the number of accesses to each level of all data streams (i.e., LHS, RHS, Output).Besides taking into account the hardware architecture and memory hierarchy of Ascend chip, our approach to quantifying memory energy is based on existing methods (Chen et al., 2016; Sze et al., 2017; Kwon et al., 2019; Yang et al., 2020a; Horowitz, 2014; Yang et al., 2017) for dataflow and energy evaluation. Given the layer parameters and memory hierarchy, it amounts to 1 Tiling: determining the tiling strategy for allocating data streams on each memory level; and 2 Data movement: specifying how data streams are reused or kept stationary to determine their access numbers.In the following, we present our method for the forward and backward passes by taking the example of a convolution layer, as convolutions are the main components of CNNs and the primary source of complexity due to their high data reuse. Denote by I, F , and O its ifmaps, filters, and ofmaps, respectively.Tiling. Since the ifmaps and filters are usually too large to be stored in buffers, the tiling strategy is aimed at efficiently transferring them to the computing unit. For the forward, denote tiling parameters of ifmaps and filters at memory IfmapsL3 (DRAM) L2 L1 L0-A L0-B L0-C EE [GBPS/: [N i , C i , H I i , W I i ], Filters : [M i , C i , H F , W F ],where H F × W F is the filter size, and the remaining notations are used as follows: N -batch size, M -output channels, C -input channels, H -height, and W -width. Tiling is to determine all these parameters, which is an NP-Hard problem (Yang et al., 2020a) . The strategy that we follow is to maximize the buffer utilization as well as near compute stationary (i.e., as much reuse as possible to reduce the number of accesses to higher levels). An iterative search over possibilities subject to memory capacity constraint provides tiling combinations of ifmaps and filters on each memory level. The tiling parameters used in this design are listed in the appendix.Data movement. For data movement, at level L0, several data stationary strategies, called dataflows, have been proposed in the literature, notably weight, input, output, and row stationary (Chen et al., 2016) . Since Ascend chip provides tripled L0 buffers, partial sums can be directly stationary in the computing cube, hence equivalent to output stationary whose implementation is described in (Du et al., 2015) . For the remaining levels, our question of interest is how to move ifmaps block[N i+1 , C i+1 , H I i+1 , W I i+1 ] and filters block [M i+1 , C i+1 , H F , W F ]from level i+1 to level i efficiently. Considering that: (i) ifmaps are reused by the filters over output channels, (ii) filters are reused over the ifmaps spatial dimensions, (iii) filters are reused over the batch dimension, (iv) ifmaps are usually very large whereas filters are small, the strategy that we follow is to keep filters stationary on level i and cycle through ifmaps when fetching them from level i + 1 as shown in ?? 2. Therein, filters and ifmaps are read block-by-block of their tiling sizes, i.e., filters block[M i , C i , H F , W F ] and ifmaps block [N i , C i , H I i , W I i ].Hence, the number of filter accesses to level i + 1 is 1 whereas the number of ifmaps accesses to level i + 1 equals the number of level-i filters blocks contained in level i + 1. Following this method, the number of accesses to memory levels of each data stream can be determined. Hence, denote by n d i the number of accesses to level i of data d, and ε i the energy cost of accessing level i, given as the inverse of energy efficiency from Table 1 . Following (Chen et al., 2016) , the energy cost of moving data d from DRAM (L3) into the cube is given as:E d = n d 3 ε 3 + n d 3 n d 2 ε 2 + n d 3 n d 2 n d 1 ε 1 + n d 3 n d 2 n d 1 n d 0 ε 0 . (4)Regarding the output partial sums, the number of accumulations at each level is defined as the number of times each data goes in and out of its lower-cost levels during its lifetime. Its data movement energy is then given as:EQUATIONAlgorithm 2: Data movement from i + 1 to i levels Input: tiling parameters of ifmaps and filters at levels i + 1 and i.1 repeat 2 read next filters block of size [M i , C i , H F , W F ] from levels i + 1 to i; 3 repeat 4 read next ifmaps block of size [N i , C i , H I i , W I i ] from levels i + 1 to i;5 let the data loaded to i be processed;6 until all ifmaps are read into level i;7 until all filters are read into level i;where factor of 2 accounts for both reads and writes and the subtraction of 1 is because we have only one write in the beginning (Chen et al., 2016) . The appendix provides more details on the number of reuses at each memory level.Backward. For the backward pass, given that ∂Loss/∂O is backpropagated from the downstream, it consists in computing ∂Loss/∂F and ∂Loss/∂I. Following the derivation of backpropagation in CNNs by (Zhang, 2016) , it is given that:EQUATIONwhere Rot π (F ) is the filter rotated by 180-degree. As a result, the backward computation structure is also convolution operations, hence follows the same process as detailed above for the forward pass. 
experiments: 
 Our design was benchmarked on different computer vision tasks: classification (using CIFAR-10 ( Krizhevsky et al., 2009) and ImageNet (Krizhevsky et al., 2012) ) and superresolution (using DIV2k (Agustsson & Timofte, 2017; Timofte et al., 2017) , Set5 (Bevilacqua et al., 2012) , Set14 (Zeyde et al., 2012) , BSD100 (Huang et al., 2015) , and Urban100 (Martin et al., 2001) ).In addition, to conceal that our Boolean Logic is advantageous for edge device learning, we explore the scenario where a pretrained model is deployed to an edge device, i.e. fine tuning. On this regard, we analyze two tasks: classification and segmentation. For classification, the trained Boolean VGG-Small architecture is fine-tuned over CIFAR-100. For segmentation, the trained Boolean ResNet18 is used as backbone on DeepLabv3 (Chen et al., 2017) and fine-tuned over the Cityscapes (Cordts et al., 2016) , and Pascal VOC 2012 (Everingham et al.) datasets.In all benchmarks, the Boolean model was built following Full-precision (Zhang et al., 2018a) 32/32 93.80 100.00 1.00 BinaryConnect (Courbariaux et al., 2015) 1/32 90.10 38.58 2.59 XNOR-Net (Rastegari et al., 2016) 1 the sketch of the baseline full-precision (FP) architecture such that its arithmetic layers are Boolean and removing FP-specific components, such as ReLU, PReLU activations or BatchNorm (unless mentioned otherwise). Following the literature (Chmiel et al., 2021) , the first and the last layers were kept in FP. Adam (Kingma & Ba, 2014) was used as the optimizer of these FP layers, while our Boolean optimizer was used on the remaining Boolean part. The full details of all experiments are provided in the supplementary material.Proof of the proposed concept was initially validated on CIFAR-10 with VGG-Small (Simonyan & Zisserman, 2014) baseline. In the experiments, our boolean architecture follows the layout of (Courbariaux et al., 2015) , except that we exclude batch normalization. This configuration obtained a top-1 accuracy of 90.29 ± 0.09% (estimated over six repetitions), showing similar performance to (Courbariaux et al., 2015) , which has 32-bit activations and is full-precision during training (see Table 2 ). Higher performances are obtained when including batch normalization after convolutions and the activation from (Liu et al., 2018) (referred to as Boolean with BN in the table), with a classification performance equal to 92.37 ± 0.01% (estimated over five repetitions) which is almost 1 point closer to the FP counterpart. Complementary comparisons with other methodologies using the modified version of VGG-Small (ending with 1 FC layer) are available in the supplementary material.In terms of energy, it is clear that our Boolean methodology is much more energy efficient for inference and training than latent-based training, which uses FP data streams.Compared to the FP network, our methodology with and without BN achieves 20.53× and 27.42× gain in energy, respectively. Notably, the use of BN provides greater network accuracy at the expense of a slight increase in energy consumption. Yet, even with BN, our methodology provides the best energy/accuracy ratio of all methods.For ImageNet classification, we use the ResNet18 (He et al., 2015) baseline to validate our Boolean methodology with and without BN. In the former case, our network follows (Liu et al., 2018) to define the basic block, whilst replacing the average pooling by a Boolean activation on downsampling blocks, see Figure 7a . The latter case, Boolean w/o BN, uses the Boolean arithmetic components and Boolean activations arranged as the FP analogues in the original down-sampling block (He et al., 2015) , see Figure 7b . Table 3 presents the results of this experiment. In the comparison, we include methodologies using the standard architecture, and also architectures with additional training strategies to increase performance. For instance, models with larger size, FP data propagation throughout the network via residuals and knowledge distillation (KD)-based learning (with a ResNet34 teacher). In addition to Boolean logic training, it is important to emphasize that our concept has streamlined Boolean architecture. In contrast, the comparative BNNs methods heavily enrich the network with fullprecision components, e.g., FP shifts (Liu et al., 2020) , FP attention weights (Guo et al., 2022) or FP scalars on binary weights (Rastegari et al., 2016) (details are provided in the supplementary material). Even deprived of such FP mechanisms, our native Boolean learning methodology achieves the best accuracy on learning with the basic architecture and without a teacher. In both modalities, our method proves to be the most favorable from the energy point of view, reducing by up to 11.21× the energy consumption of 32-bit FP training and several orders of magnitude better than the popular Bi-RealNet. These findings are paramount for training on low powerful computing devices. Further, based on approximation theory showing that reduced-precision models require a larger size to close the gap to full-precision counterparts (Elbrächter et al., 2021) , we investigated how much Boolean models need to be enlarged to reach full-precision performance. Table 3 shows that the Boolean concept recovers totally and outperforms the FP level by 4× filter enlarging, i.e., base 256, at which it still provides 2.58× energy reduction. It also outperforms PokeBNN (Zhang et al., 2022) , which uses ResNet50 as a teacher. In order to avoid additional computational burden during evaluation, the log-its predicted from the ResNet34 teacher were precomputed before launching the experiments. For completeness, we also implemented neural gradient quantization to quantize it by using INT4 quantization with logarithmic round-to-nearest approach (Chmiel et al., 2021) and statistics aware weight binning (Choi et al., 2018) . On ImageNet, we confirm that 4 bits quantization is enough to recover standard backpropagation performances (67.53% in 100 epochs, more details in the appendix).The detailed energy analysis from Section 3.2 indicates that our method consumes significantly less energy than other popular BNN methodologies (see Table 2 ). This suggests that Boolean trained networks are particularly well suited for training and fine-tuning on low capacity devices. The main idea is to take advantage of a large central server that is amenable to gather large number of samples for training a Boolean NN with low error. This pre-trained network is then downloaded by edge devices. We assume edge devices have limited computational capacity, and limited access to data. Hence, our goal is to employ a Boolean backbone, and fine-tune it locally (i.e. directly on-device).For this experiment, we aim at exploring the adaptation capabilities of our methodology to similar problems but different data. We used the VGG-Small architecture trained on CIFAR-10 and fine-tune to CIFAR-100. We also show the results when the starting model was trained on CIFAR-100 and fine-tuned to CIFAR-10. In all experiments the results are provided without BN on the architectures. This is mainly because the number of categories is 10 times larger with less labels per class, see Figure 2 . It is worth noting that, as in the FP models, the fine tuning of our Boolean models also ensures robust learning when the initial model was trained on a more complex task. We have extended the application of our novel Boolean Logic training methodology to the realm of semantic segmentation, a task that necessitates pixel-level classification and thus requires highly detailed feature extraction. This is a formidable challenge for highly quantized networks. Leveraging the success of our method in image classification, we adapted the same network architecture, to cater to the intricacies of semantic segmentation. For this experiment the baseline is the DeepLabv3 (Chen et al., 2017) network structure using as backbone the pretrained Boolean ResNet18 without BN (Figure 7b ) and followed by the Boolean Atrous Pyramid Pooling (BoolASPP) module. Our experimental findings demonstrate that this approach not only retains the inherent lightweight advantages of extremely quantized NNs, but also markedly improves the performance in complex semantic segmentation tasks.We utilized the AdamW (Loshchilov & Hutter, 2017) optimizer with an initial learning rate of 5 × 10 -4 for real parameters, and the Boolean Logic optimizer with an initial learning rate of 12 for Boolean parameters. Given the distinct characteristics of Boolean logic learning, particularly in the early stages of training, we noticed a tendency for parameters to flip easily due to large backward signals in semantic segmentation tasks. This issue is further exacerbated by the fact that each pixel in the image contributes to the backward signal, even in small batch sizes. To better preserve the integrity of the pretrained backbone, we reduced the Boolean learning rate within the backbone from 12 to 6.Throughout the process, we employed a polynomial learning rate policy with a power factor of p = 0.9 for all parameters and conducted the optimization using the cross-entropy loss function. The model was trained on Cityscapes (Cordts et al., 2016) dataset for 140K iterations, or on Pascal VOC 2012 dataset (Everingham et al.) for 80K iterations, with a batch size of 8. It is also noteworthy that we refrained from using auxiliary loss or knowledge distillation techniques, as these methods introduce additional computational burdens, which contradict our goal of efficient on-device training.As demonstrated in Table 5 , our proposed method attains a mIoU of 67.4% on the Cityscapes dataset. This performance significantly surpasses previous BNN attempts and closely approaches the efficacy of full-precision networks. Similarly, our methodology yields results close to the FP baseline on the Pascal VOC 2012 dataset. Notably, this enhancement is achieved without necessitating the intermediary use of floating-point parameters during the training process, underscoring the efficiency and efficacy of our approach.We evaluate our Boolean design capabilities to synthesize data. In order to capitalize efficiency, we used the small version of the popular EDSR method (Lim et al., 2017) for super-resolution, with eight residual blocks, later referred to as Small EDSR. Our Boolean architecture uses Boolean residual blocks as the proposed for image classification, see Figure 7b . The results with the modified FP counterpart were obtained using the official implementation2 .Table 6 summarizes the results of our experiments. The proposed Boolean Logic methodology obtains values very close to those of the FP reference at each task. It shows the best results for datasets like Set14 and BSD100. Notoriously, as is the case for the reference method, the Boolean methodology suffers significant performance reduction when the desired scaling factor is ×4. Notice that on high-resolution images like Div2K, our method generates images with high PSNR, even higher on low-resolution images like Set5. These results demonstrate that our Boolean methodology can perform pretty well on detail-demanding tasks while being considerably robust to image resolution. Table 6 : PSNR (dB) Performance of the proposed Boolean methodology for super-resolution using the EDSR baseline. 
conclusion: 
 We have presented a method for training deep neural networks that is provably efficient for resource-constrained environments. In particular, we have developed a method to estimate the energy consumption of NN training and apply it to our Boolean architectures. Our results suggest that full-precision performance can be totally recovered by enlarged Boolean models while gaining multifold complexity reduction. One can fine-tune these energy-efficient models on edge devices for specific tasks. Our experiments highlight that Boolean models can handle finer tasks, contrary to the misbelief that binary models only work for image classification. This section provides supplementary details on how the number of accesses to each memory level in Ascend chip architecture is determined by taking as example a 2D convolution layer whose parameters are summarized in Table 7 .A.1. ForwardIn the forward pass, there are three types of input data reuse:• For an H I × W I ifmap, there are H O × W O convolutions performed with a single H F × W F filter to generate a partial sum. The filter is reused H O × W O times, and this type of reuse is defined as filter convolutional reuse. Also, each feature in the ifmaps is reused H F × W F times, and this is called feature convolutional reuse.• Each ifmap is further reused across M filters to generate M output channels. This is called ifmaps reuse.• Each filter is further reused across the batch of N ifmaps. This type of reuse is called filter reuse.As mentioned in Section 3.2, a loop tiling strategy for convolutional layers is necessary in order to transmit ifmaps and filters through the memory hierarchy efficiently. Determining tiling parameters, which are summarized in I (n I i ) M M2 × α v α v 2 × α h α h 2 M2 M1 × α v 2 α v 1 × α h 2 α h 1 M1 M0 × α v 1 α v 0 × α h 1 α h 0 H F × W F × α v 0 × α h 0 F (n F i ) 1 N N2 × H O H O 2 × W O W O 2 N2 N1 × H O 2 H O 1 × W O 2 W O 1 N1 N0 × H O 1 H O 0 × W O 1 W O 0 O (n O i ) 1 1 1 1Table 9 : Numbers of accesses at different memory levels of forward convolution. calculated as:EQUATIONwhere Q I i /Q F i and b I /b F represent the memory and bitwidth of ifmaps/filters, respectively. From the obtained tiling parameters, the number of accesses that is used for (4) and ( 5) is determined by taking into account the data movement strategy as shown in ?? 2. As a result, Table 9 summarizes the number of accesses to memory levels for each data type in the forward pass. Therein,α v = H O /H I , α h = W O /W I , H O i /W O i define the height/width of tiling ofmaps in Li buffers, α v i = H O i /H I i , and α h i = W O i /W I i for i = 2, 1, and 0. 
 
 Appendix: In the backward pass, it consists in computing two gradients ∂Loss/∂F and ∂Loss/∂I. As described in Section 3.2, we can evaluate the memory energy of the backward pass by following the exact mechanism of the forward pass with the respective shape parameters. For instance, Table 10 summarizes the number of accesses at each memory level in the backward pass when calculating the gradient G I = ∂Loss/∂I. Therein, C i defines the number of tiling ifmaps in Li buffer,EQUATIONβ h i = W I i /W O i for i = 2, 1, and 0.The presented methodology and the architecture of the described Boolean NNs were implemented in Pytorch and trained on Nvidia GPUs Tesla V100. The networks thought predominantly Boolean, also contain a fraction of FP parameters that were optimized using the Adam optimizer with learning rate 10 -3 . For learning the Boolean parameters we used the Boolean optimizer. Training the Boolean networks for Image Classification was conducted with learning rates η = 150 and η = 12,Data DRAM (L3) L2 L1 L0 O (n O i ) C C2 × β v β v 2 × β h β h 2 C2 C1 × β v 2 β v 1 × β h 2 β h 1 C1 C0 × β v 1 β v 0 × β h 1 β h 0 H F × W F × β v 0 × β h 0 F (n F i ) 1 N N2 × H I H I 2 × W I W I 2 N2 N1 × H I 2 H I 1 × W I 2 W I 1 N1 N0 × H I 1 H I 0 × W I 1 W I 0 G I (n G I i ) 1 1 1 1Table 10 : Numbers of accesses at different memory levels for ∂Loss/∂I.for architectures with and without batch normalization, respectively. During the experiments, both optimizers used the cosine scheduler iterating over 300 epochs.We highlight the importance of using data augmentation techniques when training low bit-width models which otherwise would overfit with standard techniques. In addition to techniques like random resize crop or random horizontal flip, we used RandAugment, lighting (Liu et al., 2020) and Mixup (Zhang et al., 2018b) . Following (Touvron et al., 2019) , we used different resolutions for the training and validation sets. For ImageNet, the training images were 192×192 px and 224×224 px for validation images. The batch size was 300 for both sets and the cross-entropy loss was used during training.VGG-Small is found in the literature with different fully-connected FC layers. Several works take inspiration from the classic work of (Courbariaux et al., 2015) , which uses 3 FC layers. Since other BNN methodologies only use a single FC layer, Table 11 presents the results with the modified VGG-Small. (Rastegari et al., 2016) 1/1 32/32 87.4 LAB (Hou et al., 2016) 1/1 32/32 87.7 RAD (Ding et al., 2019) 1/1 32/32 90.0 IR-Net (Qin et al., 2020b) 1/1 32/32 90.4 RBNN (Lin et al., 2020a) 1/1 32/32 91.3 SLB (Yang et al., 2020b) 1/1 32/32 92.0 Ours 1/1 1/16 90.8Table 11 : Top-1 accuracy for different binary methodologies using the modified VGG-Small (ending with 1 FC layer) on the CIFAR-10 dataset.The final block design for image classification was established after iterating over two models. The Boolean blocks examined were evaluated using the ResNet18 baseline architecture and adjusting the training settings to improve performance. Figure 7 presents the preliminary designs.The Boolean Block I, Figure 3a , is similar to the original ResNet18 block in that BN operations are removed and ReLUs are replaced by the Boolean activation. This design always includes a convolution in the shortcut with spatial resolution being handled by the stride. Notice that for this block we add a Boolean activation after the Maxpool module in the baseline (also for the final baseline architecture). The Boolean Block II, Figure 3b , is composed by two stacked residual modules. For downsampling blocks we use the reshaping operation to reduce the spatial resolution and enlarge the channel dimensions both by a factor of 2. The shortcut is modified accordingly with different operations in order to guarantee similar spatial dimensions before the summation.Table 12 summarizes the results obtained with the proposed designs on ImageNet. During our experimentation, we validated the hypothesis that increasing network capacity on the convolutional layers yielded higher accuracy values. However, similar to FP CNNs, we confirmed there is a limit by which the hypothesis ceases to be true, leading to overfitting. Incorporating a et al., 2020) or channel scaling through Squeeze-and-Excitation modules (Martinez et al., 2020; Guo et al., 2022) . Figure 7 shows the basic blocks of three methodologies that perform particularly well in ImageNet. Together with BN and regular activations, those techniques not only add an additional level of complexity but also lead to heavier use of computational resources and latency delays.For comparison we also show the proposed block (Figure 7a ) used in our experiments for Image Classification, Image Segmentation and Image Super-Resolution. Our block is compact in the sense that it only includes Boolean convolutions and Boolean activations, strategically placed to keep the input and output datastreams Boolean.Our Boolean architecture is based on DeepLabv3 (Chen et al., 2017) , which has shown great success in semantic segmentation. It is proven that using dilated or atrous convolutions, which preserve the large feature maps, instead of strided convolutions is prominent for this task. In our Boolean model with ResNet-18 layout, we replace the strided convolutions in the last two ResNet layers with the non-strided version, and the dilated convolutions are employed to compensate for the reduced receptive field. Thus, the images are 8× downsampled instead of 32×, preserving small object features and allowing more information flow through the Boolean network. As shown in Figure 7 , in the Boolean basic block, a 3 × 3 convolution instead of 1 × 1 convolution is used to ensure the comparable dynamic range of pre-activations between the main pass and the short-cut. Keeping these Boolean convolutional layers non-dilated naturally allows the backbone to extract multi-scale features without introducing additional computational cost.The Atrous Spatial Pyramid Pooling (ASPP) consists of multiple dilated convolution layers with different dilation rates and global average pooling in parallel, which effectively captures multi-scale information. In the Boolean ASPP (BoolASPP), we use one 1 × 1 Boolean convolution and three 3 × 3 Boolean dilated convolution with dilation rates of {12, 24, 36} following by Boolean activation functions. The global average pooling (GAP) branch in ASPP captures image-level features, which is crucial for global image understanding as well as large object segmenting accuracy. However, in BoolASPP, as shown in Figure 9c , the Boolean input X leads to significant information loss before the global average pooling may cause performance degradation on large objects. Therefore, we keep the inputs integer for the GAP branch as demonstrated in Figure 9d . To prevent numerical instability, batch normalization is used in the GAP branch before each activation function.Using BoolASPP enhances the multi-scale feature extraction and avoids parameterized upsampling layers, e.g. transposed convolution. The model was trained on the Cityscapes dataset for 400 epochs with a batch size of 8. The AdamW optimizer (Loshchilov & Hutter, 2017) with an initial learning rate of 5 × 10 -4 and the Boolean logic optimizer with a learning rate of 12 were used respectively for real and Boolean parameters. At the early training stage, parameters could easily be flipped due to the large backward signal; thus, to better benefit from the ImageNet-pretrained backbone, we reduce the learning rate for Boolean parameters in the backbone to 6. We employed the polynomial learning rate policy with p = 0.9 for all parameters. The cross-entropy loss was used for optimization. We did not employ auxiliary loss or knowledge distillation as these training techniques require additional computational cost, which is not in line with our efficient on-device training objective.We aim to reproduce closely full-precision model performance in the semantic segmentation task with Boolean architecture and Boolean logic training. Due to the nature of the Boolean network, the common regularization method, e.g., weight decay, is not applicable. Moreover, with more trainable parameters, the Boolean network can suffer from over-fitting. In particular, as shown in Table 13 , the imbalanced dataset for semantic segmentation aggravates the situation. There is a significant performance gap for several classes which has low occurrence rate, including rider (9.5%), motor (11.2%), bus (9.5%), truck (6.9%), train (17.0%). We argue that the performance gap is due to the similarity between classes and the dataset's low occurrence rate, which is confirmed as shown in Figure 10 .Data augmentation and sampling are thus critical for Boolean model training. Regarding data augmentation, we employed multi-scale scaling with a random scaling factor ranging from 0.5 to 2. We adopted a random horizontal flip with probability p = 0.5 and color jittering. In addition, we used rare class sampling (RCS) (Hoyer et al., 2022) to avoid the model over-fitting to frequent classes. For class c, the occurrence frequency in image f c is given by:EQUATION)where N is the number of samples and y i is the set of classes existing in sample i. The sampling probability of class c is thus defined as:EQUATIONMathematical and Algorithmic Sciences Laboratory, Huawei Paris Research Center, France. Correspondence to: Van Minh Nguyen <vanminh.nguyen@huawei.com>.VGG-Small with the last FC layer mapping to 100 classes.https://github.com/sanghyun-son/EDSR-PyTorch 


Decision: The paper introduces a new approach for training neural networks efficiently using boolean logic, which is hardware friendly and energy efficient. Overall, it's an interesting read and a promising direction for energy savings. All 3 reviews received are generally positive. The authors should try to address the comments received  by the camera ready.