Title: FLEXOR: TRAINABLE FRACTIONAL QUANTIZATION

Abstract: Parameter quantization is a popular model compression technique due to its regular form and high compression ratio. In particular, quantization based on binary codes is gaining attention because each quantized bit can be directly utilized for computations without dequantization using look-up tables. Previous attempts, however, only allow for integer numbers of quantization bits, which ends up restricting the search space for compression ratio and accuracy. Moreover, quantization bits are usually obtained by minimizing quantization loss in a local manner that does not directly correspond to minimizing the loss function. In this paper, we propose an encryption algorithm/architecture to compress quantized weights in order to achieve fractional numbers of bits per weight and new compression configurations further optimize accuracy/compression trade-offs. Decryption is implemented using XOR gates added into the neural network model and described as tanh(x), which enable gradient calculations superior to the straight-through gradient method. We perform experiments using MNIST, CIFAR-10, and ImageNet to show that inserting XOR gates learns quantization/encrypted bit decisions through training and obtains high accuracy even for fractional sub 1-bit weights.

1. INTRODUCTION:

Deep Neural Networks (DNNs) demand larger number of parameters and more computations to support various task descriptions all while adhering to ever-increasing model accuracy requirements. Because of abundant redundancy in DNN models (Han et al., 2016; Frankle & Carbin, 2019; Courbariaux et al., 2015) and limited scaling technology of integrated circuits due to physical challenges, numerous model compression techniques are being aggressively studied to expedite inference of DNNs across a range of device scales. As a practical model compression scheme, parameter quantization is a popular choice because compression ratio is high and regular formats after compression enable full memory bandwidth utilization that is crucial in highly parallel computing systems.

Quantization schemes based on binary codes are gaining increasing attention since quantized weights follow specific constraints to allow simpler computations during inference. Specifically, using binary codes, a weight vector is represented as q i=1 (α i v j=1 b i,j ), where q is the number of quantization bits, v is the vector size, α is a scaling factor (α ∈ R), and b i,j is a binary code (b i,j ∈ {-1, +1}). Then, a dot product with activations is conducted as q i=1 (α i v j=1 a j b i,j ), where a j is a full-precision activation. Then, the number of multiplications is reduced from v to q (expensive floating-point multipliers are less required for inference). Moreover, even though we do not discuss a new activation quantization method in this paper, if activations are also quantized by using binary codes, then most computations are replaced with bit-wise operations (using XNOR logic and population counts) (Xu et al., 2018; Rastegari et al., 2016) . Consequently, even though representation space is constrained compared with quantization methods based on look-up tables, various inference accelerators can be designed to exploit advantages of binary codes (Rastegari et al., 2016; Xu et al., 2018) . Since a successful 1-bit weight quantization method has been demonstrated in BinaryConnect (Courbariaux et al., 2015) , advances in compression-aware training algorithms in the form of binary codes (e.g., binary weight networks (Rastegari et al., 2016) and LQ-Nets (Zhang et al., 2018) ) produce 1-3 bits for quantization while accuracy drop is modest or negligible. Fundamental investigations on DNN training mechanisms using fewer quantization bits have also been actively reported to improve quantization quality (Li et al., 2017) . Previously, binary-coding-based quantization has only permitted integer numbers of quantization bits, limiting the compression/accuracy trade-off search space, especially in the range of very low quantization bits. In this paper, we propose a flexible encryption algorithm/architecture (called "FleXOR") to enable fractional sub 1-bit numbers to represent each weight while quantizated bits are trained by gradient descent. Even though vector quntization is also a well-known scheme with high compression ratio (Stock et al., 2019) , we assume the form of binary codes. Note that the number of quantization bits can be different for each layer (e.g., Wang et al. (2019) ) to allow fractional quantization bits on average. FleXOR implies fractional quantization bits for each layer that can be quantized with different bits.

The main contributions of this paper are as follows:

XOR-based encryption of quantized bits enhances compression ratio: We encrypt quantized weights to further compress quantized DNNs applying basic principles from cryptography and digital communication.

XOR-aware training algorithm learns encrypted weights: A binary model of XOR gates is used for forward propagation and model inference. We suggest calculating gradients of XOR gates using tanh functions instead of using a straight-through estimator (STE) (Bengio et al., 2013) for efficient compression-aware training.

High model accuracy with sub 1-bit quantization: To the best of our knowledge, our work is the first to explore model accuracy under 1 bit/weight when weights are quantized based on binary codes. Such an exploration plays a significant role for mobile devices where it may be possible to sacrifice modest amounts of accuracy drop in exchange for improved energy efficiency.

2. ENCRYPTING QUANTIZED BITS USING XOR GATES:

The main purpose of FleXOR is to compress quantized bits into encrypted bits that can be reconstructed by XOR gates as shown in Figure 1 . Suppose that N out bits are to be compressed into N in bits (N out > N in ). The role of an XOR-gate network is to produce various N out -bit combinations using N in bits. In other words, in order to maximize the chance of generating a desirable set of quantized bits, the encryption scheme is designed to seek a particular property where all possible 2 Nin outcomes through decryption are evenly distributed in 2 Nout space.

A linear Boolean function, f (x), maps f : {0, 1} Nin → {0, 1} and has the form of a

1 x 1 ⊕ a 2 x 2 ⊕ • • • ⊕ a Nin x Nin where a j ∈ {0, 1} (1 ≤ j ≤ N in )

and ⊕ indicates bitwise modulo-2 addition. In Figure 1 , six binary outputs are generated through six Boolean functions using four binary inputs. Let f 1 (x) and f 2 (x) be two such linear Boolean functions using x = (x 1 , x 2 , ..., x Nin ) ∈ {0, 1} Nin . The Hamming distance between f 1 (x) and f 2 (x) is the number of inputs on which f 1 (x) and f 2 (x) differ, and defined as

EQUATION

where w H (f ) = #{x ∈ {0, 1} Nin |f (x) = 1} is the Hamming weight of a function and #{} corresponds to the size of a set (Kahn et al., 1988) . The Hamming distance is a well-known method to express non-linearity between two Boolean functions (Kahn et al., 1988) and increased Hamming In Figure 1 , y 1 is represented as x 1 ⊕ x 3 ⊕ x 4 , or equivalently a vector [1 0 1 1] denoting which inputs are selected. Concatenating such vectors, a XOR-gate network in Figure 1 can be described as a binary matrix M ⊕ ∈ {0, 1} Nout×Nin (e.g., the second row of M ⊕ is [1 1 0 0] and the third row is [1 1 1 0]). Then, decryption through XOR gates is simply represented as y = M ⊕ x where x and y are the binary inputs and binary outputs of XOR gates, and addition is 'XOR' and multiplication is 'AND' (see Appendix for more details and examples).

Encrypted weight bits are stored in 1-dimensional vector format and sliced into blocks of N in -bit size as shown in Figure 2 . Then, decryption of each slice is performed by a XOR-gate network that is shared by all slices (temporally-or spatially-shared). Depending on the quantization scheme and characteristics of layers, quantized bits may need to be scaled by a scaling factor and/or reshaped.

3. FLEXOR TRAINING ALGORITHM FOR QUANTIZATION BITS DECISION:

Once the structure of XOR gates has been pre-determined and fixed to increase the Hamming distance of XOR outputs, we find quantized and encrypted bits by adding XOR gates into the model. In other words, we want an optimizer that understands the XOR-gate network structure so as to compute encrypted bits and scaling factors via gradient descent. Note that for inference, we store binary encrypted weights (converted from real number encrypted weights) in memory and generate binary quantized weights through Boolean XOR operations. Activation quantization is not discussed in this paper.

Similar to the STE method introduced in (Courbariaux et al., 2015) , Boolean functions need to be described in a differentiable manner to obtain gradients in backward propagation. For two real number inputs x 1 and x 2 (x 1 , x 2 ∈ R to be used as encrypted weights), the Boolean version of a XOR gate for forward propagation is described as (note that 0 is replaced with -1)

EQUATION

For inference, we store sign(x 1 ) and sign(x 2 ) instead of x 1 and x 2 . On the other hand, a differentiable XOR gate for backward propagation is presented as

EQUATION

where S tanh is a scaling factor for FleXOR. Note that tanh functions are widely used to approximate Heaviside step functions (i.e., y(x) = 1 if x > 0 or 0, otherwise) in digital signal processing and S tanh can control the steepness (as explained in Appendix). In the case of consecutive XOR operations, the order of inputs to be fed into XOR gates should not affect the computation of partial gradients for XOR inputs. Therefore, as a simple extension of Eq. ( 3), a differentiable XOR gate network with n inputs can be described as

EQUATION

Then, a partial derivative of f ⊕ with respect to x i (an encrypted weight) is given as

EQUATION

Note that increasing the Hamming distance is associated with more tanh multiplications for each XOR-gate network output. Then, from Eq. ( 5), we may suffer from the vanishing gradient problem since | tanh(x)| ≤ 1. To resolve this problem, we also consider a simplified alternative partial derivative expressed as

EQUATION

Eq. ( 6) shows that when we compute a partial derivative, all XOR inputs other than x i are assumed to be binary, i.e., the magnitude of a partial derivative is then only determined by x i . We use Eq. ( 6) in this paper to calculate custom gradients of encrypted weights due to fast training computations and convergence, and use Eq. ( 2) for forward propagation.

Algorithm 1: Pseudo codes of Conv Layer with FleXOR1 

w e ∈ R (k•k•Cin•Cout)/Nout •Nin Encrypted weights M ⊕ ∈ {0, 1} Nout×Nin XOR gates (shared) α ∈ R Cout

Scaling factors for each output channel Function FleXOR Conv(input, stride, padding):

for i ← 0 to (k • k • C in • C out )/N out -1 do for j ← 1 to N out do w q i•Nout+j ← (-1) •      Nin l=1 M ⊕ j,l =1 Sign c w e i•Nin+l • (-1)     

Eq. ( 2)

W q ← Reshape(w q , [k, k, C in , C out ])

return Conv(input, W q , α, stride, padding) Convolution operation for binary codes Forward Function Sign c (x):

return sign(x) Gradient Function Sign c (x, ∇): return ∇ • (1 -tanh 2 (x • S tanh )) • S tanh

Eq. ( 6)

By training the whole network including FleXOR components with using custom gradient computation methods described above, encrypted and quantized weights are obtained in a holistic manner. FleXOR operations for convolutional layers are described in Algorithm 1, where encrypted weights (inputs of a XOR-gate network) and quantized weights (outputs of a XOR-gate network) are w e and W q . We first verify basic training principles of FleXOR using LeNet-5 on the MNIST dataset. LeNet-5 consists of two convolutional layers and two fully-connected layers (specifically, 32C5-MP2-64C5-MP2-512FC-10SoftMax), and each layer is accompanied by an XOR-gate network with N in binary inputs and N out binary outputs. The quantization scheme follows 1-bit binary code with full-precision scaling factors that are shared across weights for the same output channel number (for conv layers) or output neurons (for FC layers). Encrypted weights are randomly initialized with N (µ=0, σ 2 =0.001 2 ). All scaling factors for quantization are initialized to be 0.1 (note that if batch normalization layers are immediately followed, then scaling factors for quantization are redundant). Using the Adam optimizer with an initial learning rate of 10 -4 and batch size of 50 without dropout, Figure 3 shows training loss and test accuracy when S tanh =100, elements of M ⊕ are randomly filled with 1 or 0, and for two values of N out -10 and 20. Using the 1-bit internal quantization method and (N in , N out ) encryption scheme, one weight can be represented by (N in /N out ) bits. Hence, Figure 3 represents training results for 0.4, 0.6, and 0.8 bits per weight. Note that as for a randomly filled M ⊕ , increasing N out (and N in correspondingly for the same compression ratio) increases the Hamming distance for a pair of any two rows of M ⊕ and, hence, offers the chance to produce more diversified outputs. Indeed, as shown in Figure 3 , the results for N out =20 present improved test accuracy and less variation compared with N out =10. See Appendix for the distribution of encrypted weights at different training steps.

4. PRACTICAL FLEXOR TRAINING TECHNIQUES:

In this section, we present practical training techniques for FleXOR using ResNet-32 (He et al., 2016) on the CIFAR-10 dataset (Krizhevsky et al., 2009) . As shown in Figure 4 , since FleXOR decouples the number of bits to represent weights from the quantization scheme used, various tradeoffs between memory footprint and model accuracy are possible. We show compression results for ResNet-32 using fractional numbers as effective quantization bits, such as 0.4 and 1.2, that have not been available previously.

All layers, except the first and the last layers, are followed by FleXOR components sharing the same M ⊕ structure (thus, storage foorprint of M ⊕ is ignorable). SGD optimizer is used with momentum of 0.9 and a weight decay factor of 10 -5 . Initial learning rate is 0.1, which is decayed by 0.5 at the 150 th and 175 th epoch. As learning rate decays, S tanh is empirically multiplied by 2 to cancel out the effects of weight decay on encrypted weights. Batch size is 128 and scaling factors of α are randomly initialized between 0.01 and 0.19. q is the number of bits to represent binary codes for quantization. We provide some useful training insights below with relevant experimental results. 1) Use small N tap (such as 2): FleXOR should be able to select the best out of 2 Nin possible outputs that are randomly selected from larger 2 Nout search space. Encryption performance of XOR gates is determined by randomness of 2 Nin output candidates, and is enhanced by increasing Hamming distance that is basically achieved by large N out . Now, let N tap be the number of 1's in a row of M ⊕ . Another method to enhance encryption performance is to increase N tap so as to increase the number of shuffles (through more XOR operations) using encrypted bits to generate quantized bits such that correlation between quantized bits is reduced. Large N tap , however induces vanishing gradient problems in Eq. ( 5) or increased approximation error in Eq. ( 6). Hence, in practice, FleXOR training with small N tap converges well with high test accuracy. Studying a training algorithm to understand a complex XOR-gate network with large N tap would be an interesting research topic that is beyond the scope of this work. Subsequently, we show experimental results using N tap = 2 in the remainder of this paper.

2) Use 'tanh' rather than STE: Since forward propagation for a XOR gate only needs a sign function, the STE method is also applicable to XOR-gate gradient calculations. Another alternative method to model an XOR gate is to use Eq. (3) for both forward and backward propagation as if the XOR is modeled in an analog manner (then, real number XOR outputs are quantized through STE). We compare three different XOR modeling schemes in Figure 5 with test accuracy measured when encrypted weights and XOR gates are converted to be binary for inference. FleXOR training method shows the best result because a) sign function for forward propagation enables estimating the impact of binary XOR computations on the loss function and b) ∂(tanh) for backward propagation approximates the Heaviside step function better compared to STE. Note that limited gradients from the tanh function eliminate the need for weight clipping, which is often required for quantizationaware training schemes (Courbariaux et al., 2015; Xu et al., 2018) .

3) Optimize S tanh : S tanh controls the smoothness of the tanh function for near-zero inputs. Large S tanh employs large gradient for small inputs and, hence, results in well-clustered encrypted weight values as shown in Figure 6 . Too large of a S tanh , however, hinders encrypted weights from being finely-tuned through training. For FleXOR, S tanh is a hyper-parameter to be optimized.

4):

Learning rate and S tanh warmup: Learning rate starts from 0 and linearly increases to reach the initial learning rate at a certain epoch as a warmup. Learning rate warmup is a heuristic scheme, but being widely accepted to improve generalization capability mainly by avoiding large learning rate in the initial phase (He et al., 2018; Gotmare et al., 2019) . Similarly, S tanh starts from 5 to linearly increases to 10 using the same warmup schedule of the learning rate.

5) Try various q, N in , and N out : Using a warmup scheme for 100 epochs and learning rate decay by 50% at the 350 th , 400 th , and 450 th epoch, Figure 7 presents test accuracy of ResNet-32 with various q, N in , and N out . For q>1, different M ⊕ configurations are constructed and then shared across all layers. Note that even for the 0.4bit/weight configuration (using q=1, N in =8, and N out =20), high accuracy close to 89% is achieved. 0.8bit/weight can be achieved by two different configurations (as shown on the right side of Figure 7 ) using (q=1, N in =8, N out =10) or (q=2, N in =8, N out =20). Interestingly, those two configurations show almost the same test accuracy, which implies that FleXOR is able to provide a linear relationship between the amount of encrypted weights and model accuracy (regardless of internal configurations). In general, lowering q reduces the amount of computations with quantized weights and a high N out value is necessary for very low number of bits per weight since N in also needs to be high enough to increase the Hamming distance.

We compare quantization results of ResNet-20 and ResNet-32 on CIFAR-10 using different compression schemes in Table 1 (with full-precision activation). BWN (Rastegari et al., 2016) , Bina-ryRelax (Yin et al., 2018) , and LQ-Net (Zhang et al., 2018) propose different training algorithms for the same quantization scheme (i.e., binary codes). The main idea of these methods is to minimize quantization error and to obtain gradients from full-precision weights while the loss function is aware of quantization. Because all of quantization schemes in Table 1 uses q=1 and binary codes, the amount of computations using quantized weights is the same. FleXOR, however, allows reduced memory footprint and bandwidth which are critical to improve energy efficiency of inference engine designs (Han et al., 2016; Ahn et al., 2019) . Note that even though achieving the best accuracy for 1.0bit/weight is not the main purpose of FleXOR (e.g., XOR gate may be redundant for N in =N out ), FleXOR shows the minimum accuracy drop for ResNet-20 and ResNet-32 as shown in Table 1 , probably because a) FleXOR enables model optimization integrated with quantization and encyption units instead of considering the local quantization error and b) FleXOR calculates gradients for encrypted weights by employing 'tanh' (instead of STE) that approximates a Heaviside step function.

5. EXPERIMENTAL RESULTS ON IMAGENET:

In order to show that FleXOR principles can be extended to larger models, we choose ResNet-18 on ImageNet (Russakovsky et al., 2015) . We use SGD optimizer with momentum of 0.9 and initial learning rate of 0.1. Batch size is 128, weight decay factor is 10 -5 , and S tanh is 10. Learning rate is reduced by half at the 70 th , 100 th , and 130 th . For warmup, during initial ten epochs, S tanh and learning rate increase linearly from 5 and 0.0, respectively, to initial values.

Table 2 shows the comparison on model accuracy of ResNet-18 when weights are compressed by quantization (and additional encryption by FleXOR) while activations maintain full precision. Training the entire model for ImageNet including FleXOR components is successfully performed. In Table 2 , BinaryRelax and BWN do not modify the underlying model architecture, while ABC-Net introduces a new block structure of the convolution for quantized network designs. FleXOR achieves the best top-1 accuracy even with only 0.8bit/weight and demonstrates improvement of model accuracy as the number of bits per weight increases. Refer to Appendix for the accuracy graph with q=1 and more results with q=2.

6. CONCLUSION:

This paper proposes an encryption algorithm/architecture, FleXOR, as a framework to further compress quantized weights. Encryption is designed to produce more outputs than inputs by increasing the Hamming distance of output functions when output functions are linear functions of inputs. Output functions are implemented as a combination of XOR gates which are included in the model to find encrypted and quantized weights through gradient descent while using the tanh function for backward propagation. FleXOR is able to provide fractional numbers of bits for weights and, thus, much wider trade-offs between weight storage and model accuracy. Experimental results show that ResNet on CIFAR-10 and ImageNet can be represented by sub 1-bit/weight compression with high accuracy.

A APPENDIX A.1 EXAMPLE OF A XOR-GATE NETWORK STRUCTURE REPRESENTATION:

In Figure 1 , outputs of a XOR-gate network are given as

y 1 = x 1 ⊕ x 3 ⊕ x 4 y 2 = x 1 ⊕ x 2 y 3 = x 1 ⊕ x 2 ⊕ x 3 y 4 = x 3 ⊕ x 4 y 5 = x 2 ⊕ x 4 y 6 = x 2 ⊕ x 3 ⊕ x 4 .

Equivalently, the same structure as above can be represented in a matrix as

EQUATION

Note that elements of M ⊕ are matched with coefficients of y i (1 ≤ i ≤ 6). For two vectors y = {y 1 , y 2 , y 3 , y 4 , y 5 , y 6 } and x = {x 1 , x 2 , x 3 , x 4 }, the following equation holds:

EQUATION

where element-wise addition and multiplication are performed by 'XOR' and 'AND' function, respectively. In Eq. ( 7), N tap (i.e., the number of '1's in a row) is 2 or 3.

A.2 SUPPLEMENTARY DATA FOR BASIC FLEXOR TRAINING PRINCIPLES:

A Boolean XOR gate can be modeled as F ⊕ (x 1 , x 2 ) = (-1) sign(x 1 ) sign(x 2 ) if 0 is replaced with -1 as shown in Table 3 .

sign(x 1 ) sign(x 2 ) F ⊕ (x 1 , x 2 ) -1 -1 -1 -1 +1 +1 +1 -1 +1 +1 +1 -1

Table 3 : An XOR gate modeling using F ⊕ (x 1 , x 2 ).

In Eq. ( 7), forward propagation for y 3 is expressed as

EQUATION

)

while partial derivative of y 3 with respect to x 1 is given as (not derived from Eq. ( 9))

EQUATION

or as

EQUATION

We choose Eq. ( 11), instead of Eq. ( 10), as explained in Section 3.

As shown in Figure 8 , large S tanh yields sharp transitions for near-zero inputs. Such a sharp approximation of the Heaviside step function produces large gradient values for small inputs and encourages encrypted weights to be separated into negative or positive values. Too large S tanh , however, has the same isseus of a too large learning rate.

Under review as a conference paper at ICLR 2020 In this subsection, we additionally provide various graphs and accuracy tables for ResNet models on CIFAR10 and ImageNet. We also present experimental results from wider hyper-parameters searches including q = 2 with two separate M ⊕ configurations (with the same N in and N out for two M ⊕ matrices). (a) Initial Learning Rate (0.1): Test accuracy of ResNet-32 on CIFAR10 using the learning schedule in Figure 5 and various initial learning rates (0.05, 0.1, 0.2, 0.5). 5 . As for weight clipping, we restrict the encrypted weights to be ranged as (-2.0/S tanh , +2.0/S tanh ). As can be observed, the red line implies that weight clipping is not effective with FleXOR. (b) Test accuracy using q = Compared to the above plots (Figure 13a ), this figure shows that a combination of multiple M ⊕ for a binary code can lead to stable learning curves and higher model accuracy. Table 5 : Weight compression comparison of ResNet-20 and ResNet-32 on CIFAR-10 using learning rate warmup (for 100 epochs) and q = 2. As mentioned in Figure 4 , multiple M ⊕ can be combined for multi-bit quantization schemes. Then, the number of scaling factors should be doubled. FleXOR with q = 2 and two different M ⊕ structures achieve full-precision accuracy when both N in and N out are 10. 

-2 -1 0 1 2 y = tanh(x • S tanh ) -1.0 -0.5 0.0 0.5 1.0 S tanh =1 S tanh =2 S tanh =5 S tanh =10 S tanh =20 -2 -1 0 1 2 y = S tanh (1 -tanh 2 (x • S tanh ))

:

Kernel size is k × k, the number of input channel and output channel are Cin and Cout, respectively

Reported by BinaryRelax paper(Yin et al., 2018)

Reported by LQ-Net paper(Zhang et al., 2018)

Based in https://github.com/akamaster/pytorch resnet cifar10, where we replaced the ReLU functions with leaky ReLU functions.

Figures and Tables:

Figure 1: FleXOR components added to the quantized DNNs to compress quantized weights through encryption. Encrypted weight bits are decrypted by XOR gates to produce quantized weight bits.

Figure 3: Test accuracy and training loss (average of 6 runs) with LeNet-5 on MNIST when M ⊕ is randomly filled with {0, 1}. N out is 10 or 20 to generate, effectively, 0.4, 0.6, or 0.8 bit/weight quantization.

Figure 5: Test accuracy comparison on ResNet-32 (for CIFAR-10) using various XOR training methods. N out = 10, N in = 8, q = 1 (thus, 0.8bit/weight), and S tanh = 10.

Figure7: Test accuracy of ResNet-32 on CIFAR10 using learning rate warmup and various q, N in , and N out . The results on the right side are obtained by 5 runs.

Figure 8: The left graph shows hyperbolic tangent (y = tanh(x • S tanh )) graphs with various scaling factors (S tanh ), . The right graph shows their derivatives. These graphs support the arguments of 'Optimize S tanh ' in Section 4.

Figure 9: Test accuracy and training loss of LeNet-5 on MNIST when number of '1's in each row of M ⊕ is fixed to be 2 (N tap = 2). N out is 10 or 20 to generate, effectively, 0.4, 0.6, or 0.8 bit/weight quantization. With low N tap of M ⊕ , MNIST training presents less variations on training loss and test accuracy that in Figure 3.

Figure 9 presents training loss and test accuracy when N tap = 2 and N out is 10 or 20. Compared with Figure 3, N tap = 2 presents improved accuracy for the cases of high compression configurations (e.g., N in = 4 and N out = 10). We use N tap = 2 for CIFAR-10 and ImageNet, since low N tap avoids gradient vanishing problems or high approximation errors in Eq.(5) or Eq.(6).

Figure10plots the distribution of encrypted weights at different training steps when each row of M ⊕ is randomly assigned with {0, 1} (i.e., N tap is N in /2 on average) or assigned with only two 1's (N tap =2). Due to gradient calculations based on tanh and high S tanh , encrypted weights tend to be clustered on the the left or right (near-zero encrypted weights become less as N tap increases) even without weight clipping.

Figure 10: Distribution of encrypted weight values for FC1 layer of LeNet-5 at different training steps using S tanh = 100 and N out = 10. (Left): M ⊕ is randomly filled (N tap ≈ N in /2). (Right): N tap = 2 for every row of M ⊕ .

No Weight Clipping: Test accuracy of ResNet-32 on CIFAR10 using the learning schedule in Figure

weight decay(10 -5 ) (c) Weight Decay Factor (10 -5 ): Two graphs depict test accuracy of ResNet-18 on ImageNet with or without weight decay. The learning rate in the red line (no weight decay) is reduced by half at the 100 th , 130 th and 150 th epochs. The learning rate of the blue line (with weight decay) is reduced by half at 70 th , 100 th and 130 th epochs. With weight decay (blue graph), despite slow convergence in the early training steps, model accuracy is eventually higher than the red one without weight decay scheme.

Figure 12: Comparison of various hyper-parameter choices for CIFAR-10 or ImageNet.

Figure 14: Test accuracy (Top-1) of ResNet-18 on ImageNet using FleXOR.

Weight compression comparison of ResNet-20 and ResNet-32 on CIFAR-10. For FleXOR, we use warmup scheme, S tanh =10, and N out =20. All of quantization schemes in the table follow the form of binary codes with q=1, and hence, the amount of computations becomes the same. FleXOR, however, saves on/off-chip memory requirements further becasue N in /N out is less than 1.

Table content: <table><tr><td/><td/><td>ResNet-20</td><td/><td/><td>ResNet-32</td><td/></tr><tr><td/><td>FP</td><td>Compressed</td><td>Diff.</td><td>FP</td><td>Compressed</td><td>Diff.</td></tr><tr><td colspan="2">BWN (1 bit) 92.68% 2</td><td>87.44%</td><td colspan="2">-5.24% 93.40% 2</td><td>89.49%</td><td>-4.51%</td></tr><tr><td colspan="2">BinaryRelax (1 bit) 92.68% 2</td><td>87.82%</td><td colspan="2">-4.86% 93.40% 2</td><td>90.65%</td><td>-2.80%</td></tr><tr><td colspan="2">LQ-Net (1 bit) 92.10% 3</td><td>90.10%</td><td>-1.90%</td><td>-</td><td>-</td><td>-</td></tr><tr><td>FleXOR (1.0 bit)</td><td/><td>90.44%</td><td>-1.47%</td><td/><td>91.36%</td><td>-0.97%</td></tr><tr><td>FleXOR (0.8 bit) FleXOR (0.6 bit)</td><td>91.87% 4</td><td>89.71% 89.04%</td><td>-2.10% -2.83%</td><td>92.33% 4</td><td>90.78% 90.30%</td><td>-1.55% -2.03%</td></tr><tr><td>FleXOR (0.4 bit)</td><td/><td>87.85%</td><td>-4.02%</td><td/><td>89.61%</td><td>-2.72%</td></tr></table>

Weight compression comparison of ResNet-18 on ImageNet using various compression schemes. All of quantization schemes in the table follow the form of binary codes with q=1, and hence, the amount of computations becomes the same. FleXOR, however, saves on/off-chip memory requirements further becasue N in /N out is less than 1.

Table content: <table><tr><td>Methods</td><td colspan="3">Bits/Weight Top-1 Top-5 Storage Saving</td></tr><tr><td>Full Precision (He et al., 2016)</td><td>32</td><td>69.6% 89.2%</td><td>1×</td></tr><tr><td>BWN (Rastegari et al., 2016)</td><td>1</td><td>60.8% 83.0%</td><td>∼ 32×</td></tr><tr><td>ABC-Net (Lin et al., 2017)</td><td>1</td><td>62.8% 84.4%</td><td>∼ 32×</td></tr><tr><td>BinaryRelax (Yin et al., 2018)</td><td>1</td><td>63.2% 85.1%</td><td>∼ 32×</td></tr><tr><td/><td>0.8</td><td>63.8% 84.8%</td><td>∼ 40×</td></tr><tr><td>FleXOR (N out = 20)</td><td>0.6</td><td>62.0% 83.7%</td><td>∼ 53×</td></tr><tr><td/><td>0.5</td><td>61.3% 83.1%</td><td>∼ 64×</td></tr></table>

Figure 13: Test accuracy of ResNet-32 on CIFAR-10 using learning rate warmup (for 100 epochs) and N out = 20Under review as a conference paper at ICLR 2020 Weight compression comparison of ResNet-20 and ResNet-32 on CIFAR-10 when N out = 10. Parameters and recipes not described in the table are the same as in Table1. We also present compression ratio for fractional quantized ResNet-32 when one scaling factor (α) is assigned to each output channel.

Table content: <table><tr><td colspan="2">Bits/Weight</td><td colspan="2">ResNet-20</td><td colspan="2">ResNet-32</td><td>Comp. Ratio</td></tr><tr><td>FP</td><td>32</td><td>91.87%</td><td>-</td><td>92.33%</td><td>-</td><td>1.0x</td></tr><tr><td>N in =10, N out =10</td><td>1.0</td><td colspan="4">90.21% -1.66% 91.40% -0.93%</td><td>29.95×</td></tr><tr><td>N in =9, N out =10</td><td>0.9</td><td colspan="4">90.03% -1.84% 91.28% -1.05%</td><td>31.82×</td></tr><tr><td>N in =8, N out =10</td><td>0.8</td><td colspan="4">89.59% -2.28% 90.86% -1.47%</td><td>35.32×</td></tr><tr><td>N in =7, N out =10</td><td>0.7</td><td colspan="4">89.24% -2.63% 90.54% -1.79%</td><td>39.68×</td></tr><tr><td>N in =6, N out =10</td><td>0.6</td><td colspan="4">89.21% -2.66% 90.41% -1.92%</td><td>45.27×</td></tr><tr><td>N in =5, N out =10</td><td>0.5</td><td colspan="4">88.54% -3.33% 89.82% -2.51%</td><td>52.70×</td></tr><tr><td/><td/><td>ResNet-20</td><td/><td/><td colspan="2">ResNet-32</td></tr><tr><td/><td>FP</td><td>Quant.</td><td>Diff.</td><td>FP</td><td>Quant.</td><td>Diff.</td></tr><tr><td colspan="7">TWN (ternary) 92.68% 88.65% -4.03% 93.40% 90.94% -2.46%</td></tr><tr><td colspan="7">BinaryRelax (ternary) 92.68% 90.07% -1.91% 93.40% 92.04% -1.36%</td></tr><tr><td colspan="7">TTQ (ternary) 91.77% 91.13% -0.64% 92.33% 92.37% +0.04%</td></tr><tr><td colspan="4">LQ-Net (2 bit) 92.10% 91.80% -0.30%</td><td>-</td><td>-</td><td>-</td></tr><tr><td colspan="2">FleXOR(q = 2, N out = 20)</td><td/><td/><td/><td/><td/></tr><tr><td>N in =20, 2.0 bit/weight</td><td/><td colspan="2">91.28% -0.59%</td><td/><td colspan="2">91.99% -0.34%</td></tr><tr><td>N in =18, 1.8 bit/weight</td><td/><td colspan="2">90.96% -0.91%</td><td/><td colspan="2">92.01% -0.32%</td></tr><tr><td>N in =16, 1.6 bit/weight</td><td>91.87%</td><td colspan="2">90.81% -1.06%</td><td>92.33%</td><td colspan="2">91.55% -0.78%</td></tr><tr><td>N in =14, 1.4 bit/weight</td><td/><td colspan="2">90.22% -1.65%</td><td/><td colspan="2">91.33% -1.00%</td></tr><tr><td>N in =12, 1.2 bit/weight</td><td/><td colspan="2">89.88% -1.99%</td><td/><td colspan="2">90.78% -1.55%</td></tr><tr><td colspan="2">FleXOR(q = 2, N out = 10)</td><td/><td/><td/><td/><td/></tr><tr><td>N in =10, 2.0 bit/weight</td><td/><td colspan="2">91.06% -0.81%</td><td/><td colspan="2">92.61% +0.28%</td></tr><tr><td>N in =9, 1.8 bit/weight</td><td/><td colspan="2">91.00% -0.87%</td><td/><td colspan="2">92.09% -0.24%</td></tr><tr><td>N in =8, 1.6 bit/weight</td><td>91.87%</td><td colspan="2">90.82% -1.05%</td><td>92.33%</td><td colspan="2">91.96% -0.37%</td></tr><tr><td>N in =7, 1.4 bit/weight</td><td/><td colspan="2">90.81% -1.06%</td><td/><td colspan="2">91.63% -0.70%</td></tr><tr><td>N in =6, 1.2 bit/weight</td><td/><td colspan="2">90.28% -1.59%</td><td/><td colspan="2">91.32% -1.01%</td></tr></table>

Peer review (iclr2020):
The paper proposes a new approach for quantizing neural networks. It looks at binary codes for quantization instead of the usual lookup tables because quantization approaches that rely on lookup tables have the drawback that during inference, the network has to fetch the weights from the lookup table and work with the full precision values, which means that the computation cost is remains the same as the non-quantized network. The paper presents FleXOR gates, a fast and efficient logic gate for mapping bit sequences to binary weights. The benefit of the gate is that the bit sequence can be shorter than the binary weights which means that the code length can be less than 1 bit per weight. The paper also proposes an algorithm for end-to-end training of the quantized neural networks. It proposes the use of tanh to approximate the non-differentiable Heaviside step function in the backward pass. Novelty The idea of using logic gates for dequantization is interesting and (as far as I know) novel. One can imagine, that specialized hardware build on this idea could very efficient for inference (in terms of energy cost). Writing The paper is very well written and completed with great visualizations and pseudocode. Kudos to the authors, I really enjoyed reading it. However, I do not think it is justified to go over the 8 page soft limit. I would recommend that the authors perhaps shorten section 3 or remove figure 9 to fit it into 8 pages. Significance/Impact The paper is motivated by the high computation cost of working with full precision values. But this paper also works with full precision weights, since it has a full precision scaling factor (alpha) and, as far as I understood, works with full precision values during forward propagation. This means that there likely are no computational savings when compared to lookup tables. The evaluation section lacks experiments that evaluate the computational savings. The baselines should include quantization methods based on lookup tables, and there should be a comparison of computational costs. The baselines that are presented (BWN etc.) offer a tradeoff between accuracy and computational costs, yet they are only compared in accuracy. I would strongly recommend including the computational cost of each method in the evaluation section. Overall assessment: While I enjoyed reading this paper, I am leaning towards rejection due to the shortcomings of the evaluation section.