FleXOR: Trainable Fractional Quantization
Authors: Dongsoo Lee
Se Jung Kwon
Byeongwook Kim
Yongkweon Jeon
Baeseong Park
Jeongin Yun
Gu-Yeon Wei
Venue:  (2020)
Keywords: Quantization
Model Compression
Trainable Compression
XOR
Encryption

Abstract: Parameter quantization is a popular model compression technique due to its regular form and high compression ratio. In particular, quantization based on binary codes is gaining attention because each quantized bit can be directly utilized for computations without dequantization using look-up tables. Previous attempts, however, only allow for integer numbers of quantization bits, which ends up restricting the search space for compression ratio and accuracy. Moreover, quantization bits are usually obtained by minimizing quantization loss in a local manner that does not directly correspond to minimizing the loss function. In this paper, we propose an encryption algorithm/architecture to compress quantized weights in order to achieve fractional numbers of bits per weight and new compression configurations further optimize accuracy/compression trade-offs. Decryption is implemented using XOR gates added into the neural network model and described as $\tanh(x)$, which enable gradient calculations superior to the straight-through gradient method. We perform experiments using MNIST, CIFAR-10, and ImageNet to show that inserting XOR gates learns quantization/encrypted bit decisions through training and obtains high accuracy even for fractional sub 1-bit weights.

Under review as a conference paper at ICLR 2020
FLEXOR: TRAINABLE FRACTIONAL QUANTIZATION
Anonymous authors Paper under double-bling blind review
ABSTRACT
Parameter quantization is popular model compression technique due to its reg- ular form and high compression ratio. In particular, quantization based on bi- nary codes is gaining attention because each quantized bit can be directly utilized for computations without dequantization using look-up tables. Previous attempts, however, only allow for integer numbers of quantization bits, which ends up re- stricting the search space for compression ratio and accuracy. Moreover, quanti- zation bits are usually obtained by minimizing quantization loss in local manner that does not directly correspond to minimizing the loss function. In this paper, we propose encryption algorithn/architecun to compress quantized weights in order to achieve fractional numbers of bits per weight and new compression con- figurations further optimize accuracy/compression trade-off'. Decryption is im- plemented using XOR gates added into the neural network model and described as (anh(x), which enable gradient calculations superior the straight-through gradi- ent method. We perform experiments using MNIST, CIFAR-10, 10, and ImageNet to show that inserting XOR gates learns quantization/eneryptee bit decisions through training and obtains high accuracy even for fractional sub 1-bit weights.
INTRODUCTION
Deep Neural Networks (DNNs) demand larger number of parameters and more computations to support various task descriptions all while adhering to ever-increasing model accuracy requirements. Because of abundant redundancy in DNN models (Han et al., 2016 Frankle & Carbin 2019 Cour- bariaux et al.) (015) and limited scaling technology of integrated circuits due to physical challenges, numerous model compression techniques are being aggressively studied to expedite inference of DNNs across a range of device scales. As a practical model compression scheme, parameter quanti- zation is a popular choice because compression ratio is high and regular formats after compression enable full memory bandwidth utilization that is crucial in highly parallel computing systems.
Quantization schemes based binary codes gaining increasing attention since quantized weights follow specific constraints to allow simpler computations during inference. Specifically, using binary codes, weight vector is represented as Di_1(a: Ey=1 bi.j), where q is the number of quantization bits, is the vector size, is scaling factor (a E IR), and buj is binary code (bi.) E +1). Then, a dot product with activations is conducted as Σi (a; where 0j is full-precision activation. Then, the number of multiplications is reduced from o to q (expensive floating-point multipliers are less required for inference). Moreover, even though do not discuss a new activation quantization method in this paper, if activations are also quantized by using binary codes, then most computations are replaced with bit- operations (using XNOR logic and population counts) (Xu et al. 2018 Rastegari et al.) 2016). Consequently, even though representation space is constrained compared with quantization methods based on look-up tables, various inference accelerators can be designed to exploit advantages of binary codes (Rastegari et al.) 2016 Xu et al. 2018). Since a successful 1-bit weight quantization method has been demonstrated in BinaryCCnnect (Courbariaux et al. 2015), advances in compression-aware training algorithms in the form of binary codes (c.g., binary weight networks (Rastegari et al., 2016) and LQ-Nets (Zhang let al. 2018) produce 1-3 bits for quantization while accuracy drop is modest negligible. Fun- damental investigations on DNN training mechanisms using fewer quantization bits have also been actively reported to improve quantization quality (Li et al. 2017).

---

Under review as a conference paper at ICLR 2020
FleXOR Components
Layer N+1
>>
Encrypted Weights (Binary)
Quantized Weights
Xi
> Y2
Xg
X9
gaRe Network
X
Layer N
X1
1>9
> y6
Encrypted
Quantized
>
Weights (Binary)
Weights
Decryption using (N,=4, N6. LORe Network
Figure 1: FleXOR components added the quantized DNNs to compress quantized weights through eneryption. Encrypted weight bits are decrypted by XOR gates to produce quantized weight bits.
Previously, binary-coding- -baed quantization has only permitted integer numbers of quantization bits, limiting the compression/accuracy trade-off search space, especially in the range of very low quantization bits. In this paper, we propose a flexible encryption algorithm/archiiccuue (called "FleXOR") to enable fractional sub 1-bit numbers to represent each weight while quantizated bits are trained by gradient descent. Even though vector quntization is also a well-known scheme with high compression ratio (Stock et al. 2019). we assume the form of binary codes. Note that the number of quantization bits can be different for each layer (e.g.. Wang et al. (2019)) to allow fractional quantization bits on average. FleXOR implies fractional quantization bits for each layer that can be quantized with different bits.
The main contributions of this paper are as follows:
XOR- encryption of quantized bits enhances compression ratio: We encrypt quantized weights further compress quantized DNNs applying basic principles from eryptography and dig- ital communication.
XOR-aware training algorithm learns encrypted weights: binary model of XOR gates used for forward propagation and model inference. We suggest calculating gradients of XOR gates using tanh functions instead using a straight- estimator (STE) (Bengio et al. 2013 for efficient compression-awe aware training.
High model accuracy with sub 1-bit quantization: To the best of our knowledge, our work is the first explore model accuracy under 1 bit/weight when weights are quantized based binary codes. Such an exploration plays a significant role for mobile devices where it may be possible to sacrifice modest amounts of accuracy drop in exchange for improved energy efficiency.
2 ENCRYPTING QUANTIZED BITS USING XOR GATES
The main purpose of FleXOR is to compress quantized bits into encrypted bits that can be recon- structed by XOR gates as shown in Figure [ Suppose that Nowt bits are to be compressed into Ni, bits (N > N..). The role of XOR-gate network is to produce various Nout-bit combinations using Nin bits. other words, in order to maximize the chance of generating desirable set of quantized bits, the encryption scheme is designed to seek particular property where all possible No, outcomes through decryption are evenly distributed in Nout space.
A linear Boolean function, fw)) maps f: (0,1), N:.. - 1). and has the form of a @ ag22 O ... @ ONN,, &. where ay E {0, 1) (1 S < N,,) and @ indicates bitwise modulo-2 addition. In Figure 1 six binary outputs generated through six Boolean functions using four binary inputs. Let fi (æ) and f2(x)) be two such linear Boolean functions using x = (œ1, 22 ..., IN...) E {0, No,,
The Hamming distance between (x) and f2(2) the number of inputs on which fi (xx) and f2( differ, and defined as
(1) is the Hamming weight of function and # ( cor- responds to the size of a set (Kahn et al. 1988). The Hamming distance is a well-known method to express non-linearity between two Boolean functions (Kahn et al. 1988) and increased Hamming
drr f2) :- wh( @ f2) = ( E (0, 10.1) (z) + f2(m)),
where (f( #(x {0. No.
No. f((z)
2

---

Under review as a conference paper at ICLR 2020
Encrypted Weight Vector
Slicing
Nv bits
bits
No bits
No. bits
bits
Slices of Encrypted Weights
One XOR- Network is Temporally- or Spatially-Shared
Nau bits
Nu. bits
Noj
Slices of Quantized Weights
Scaling Factor (for Binary-Code Dequantization)
Full-Precision Weights
Reshaping (2D for FC, 4D for Conv)
Encrypted weight bits are sliced and reconstructed by XOR-gate network which can be shared (in time or space). Then quantized bits after XOR gates are finally reshaped.
Figure 2:
distance between pair of two Boolean functions results in variety of outputs produced by XOR gates. Increasing Hamming distance is a required feature for cryptography to derive complicated en- cryption structure such that inverting encrypted data becomes difficult. For digital communication, the Hamming distance between encoded signals is closely related to the amount of error correction possible.
In Figure 1 is represented as æ @ O L4, or equivalently a vector [1011 denoting which inputs are selected. Concatenating such vectors. -OR-gate network in Figure I can be described as a binary matrix M E {0, 1 x N., (c.g.. the second row of M is [1 1100 and the third row 1 0)) Then, decryption through XOR gates is simply represented as y = M M where x and are the binary inputs and binary outputs of XOR gates, and addition is 'XOR' and multiplication is 'AND' (see Appendix for more details and examples).
Encrypted weight bits are stored in 1-dimensional vector format and sliced into blocks of N size as shown in 2 Then, decryption of each slice is performed by a XOR-gate network that is shared by all slices (temporally- or spatially-shared) Depending on the quantization scheme and characteristics of layers, quantized bits may need to be scaled by a scaling factor and/or reshaped.
3 FLEXOR TRAINING ALGORITHM FOR QUANTIZATION BITS DECISION
Once the structure of XOR gates has been pre-determined and fixed to increase the Hamming dis- tance of XOR outputs, we find quantized and encrypted bits by adding XOR gates into the model. In other words, want an optimizer that understands the XOR-gate network structure as com- pute encrypted bits and scaling factors via gradient descent. Note that for inference, we store binary encrypted weights (converted from real number encrypted weights) in memory and generate binary quantized weights through Boolean XOR operations. Activation quantization is not discussed in this paper.
Similar to the STE method introduced in (Courbariaux et al. 2015). Boolean functions need to be described in a differentiable manner to obtain gradients in backward propagation. For two real number inputs and 22 E I to be used as encrypted weights), the Boolean version of XOR gate for forward propagation is described as (note that is replaced with -1)
F (z1, 222 = (- sign(x1) sign(x2).
(2)
For inference, we store sign(x1) and sign (22) instead of xı and 22. On the other hand, differen- tiable XOR gate for backward propagation is presented
0 22) (-1) tanh((; Stant) tanh(x2 Stant),
where Stank is scaling factor for FleXOR. Note that tanh functions are widely used to approximate Heaviside step functions (i.e., y(2)
1 if > 0 O. otherwise) in digital signal processing and Stanh can control the steepness (as explained in Appendix). In the case of consecutive XOR operations, the order of inputs to be fed into XOR gates should not affect the computation of partial
3

---

Under review as a conference paper at ICLR 2020
gradients for XOR inputs. Therefore, as simple extension of Eq. (3), differentiable XOR gate network with n inputs can described as
(I), 02, 22, Fn) = ( (-1)"= tanh(21 Stank) tanh(22 Stanh) ... tanh Stanh):
(4)
Then, a partial derivative of g with respect to x; (an encrypted weight) is given as
a @ (x) 22..............................
[[= tanh(ay Stanh)
Stanh ( (-1) 11 tanh (xze Stanh))
(5)
dr,
tanh( Stann)
Note that increasing the Hamming distance associated with more tanh multiplications for each XOR-gate network output. Then, from Eq. (5), may suffer from the vanishing gradient problem
since tanh(x)] & 1. resolve this problem, we also consider simplified alternative partial derivative expressed
0 th 02, Inn
Stank (-1n (a, Stanh)) )) II sign(
(6)
dr,
j
Eq. (6) shows that when compute partial derivative, all XOR inputs other than x, assumed to be binary, i.c. the magnitude of a partial derivative is then only determined by z;. We use Eq. (6) in this paper to calculate custom gradients of encrypted weights due to fast training computations and convergence, and use Eq. for forward propagation.
Algorithm 1: Pseudo codes of Conv Layer with FleXOR wf R [K- Cout)/Nout -.N...
Encrypted weights XOR gates (shared)
M E 10, Nour Nin
a E Cou.
Scaling factors for each output channel
Function FleXOR Conv (input, stride, padding): : for i + to [(k k- Cin Cout)/Nout do
for j + 1 to Nout do
Nin
w Noucts t (-1)-
Sign (winNatt) (-1)
b- Eq. (
n + Reshape(w", [t., k, Cins Cour))
return Conv(inpul, W az stride, padding)
D Convolution operation for binary codes
Forward Function Sign (x)
return szgn(x)
Gradient Function Sign (x. V): return 71 tanh? (x Stant)) Stanh
Eq. 6
By training the whole network including FleXOR components with using custom gradient compu- tation methods described above, encrypted and quantized weights obtained in holistic manner. FleXOR operations for convolutional layers are described in Algorithm 1, where encrypted weights (inputs of XOR-gate network) and quantized weights (outputs of XOR-gate network) wf and W We first verify basic training principles of FleXOR using LcNet-5 on the MNIST dataset. LeNet-5 consists of two convolutional layers and two fully-connected layers (specifically, 32C5- MP2-64CC5MMP2-1111 12FC-10SoftMaxx and each layer is accompanied by an COR-gate network with Nm, binary inputs and Nout binary outputs. The quantization scheme follows 1-bit binary code with full-precision scaling factors that are shared across weights for the same output channel number (for conv layers) or output neurons (for FC layers). Enerypted weights are randomly initialized with N(u=0, 02=0.0013) All scaling factors for quantization are initialized to be 0.1 (note that if batch normalization layers are immediately followed, then scaling factors for quantization are redundant).
Kereel size & the number of input channel and output channel Cin and Cout, respectively
4

---

Under review as a conference paper at ICLR 2020
99.2
0.3
99.2
0.3
98.8
0.2
98.8
0.2
Test Accuracy
.8..5
0.2
1117a Los
Tost. Accuracy
98.3
0.2
98.1
0.2
98.1
5u= No
97.7
No
97.7
Nu. =2, Nou
0.1
110
No = 16, Nour
97.4
0.1
97.4
0.1
97.0
0.0
97.0
0.0
5000
10000
15000
20000
5000
10000 Global Step
15000
20000
Global Step
Figure 3: Test accuracy and training loss (average of runs) with LeNet-5 on MNIST when M is randomly filled with {0, 1}. Nout is 10 or 20 to generate, effectively, 0.4, 0.6, or 0.8 bit/weight quantization.
==1, N 18, 22
= 3, Nn= 2o
20 weights
20 weights
k a
K a2
K
20 bits
20 bits
20 bits
20 bits
18 bits
6 bits (Encrypted)
6 bits 1 (Encrypted)
6 bits (Encrypted)
(Encrypted)
0.9 bits/weight (1 bit quantization)
0.9 bits/weight
(3-bit quantization)
Figure 4: Using the same weight storage footprint, FleXOR enables various internal quantization schemes. (Left): 1-bit internal quantization. (Right): 3-bit internal quantization with different M configurations.
Using the Adam optimizer with an initial learning rate of 10 and batch size of 50 without dropout, shows Figure shows training loss and test accuracy when Stath=100 elements of M are randomly filled with 1 or O, and for two values of Nout - 10 and 20. Using the 1-bit internal quantization method and (Ni,, Nout)) encryption scheme, one weight can be represented by (Nu./Nuut) bits. Hence, Figure represents training results for 0.4, 0.6, and 0.8 bits weight. Note that as for randomly filled M increasing Nout (and Ni, correspondingly for the same compression ratio) increases the Hamming distance for pair of any two rows of M and, hence, offers the chance to produce more diversified outputs. Indeed, as shown in Figure 3 the results for Nour=20 present improved test accuracy and less variation compared with 1ow==10. See Appendix for the distribution of encrypted weights at different training steps.
PRACTICAL FLEXOR TRAINING TECHNIQUES
In this section, we present practical training techniques for FleXOR using ResNet-32 (He et al. 2016) on the CIFAR-10 10 dataset (Krizhevsky et al.) 2009). As shown in Figure 4 since FleXOR decouples the number of bits to represent weights from quantization scheme used, various trade- offs between memory footprint and model accuracy possible. We show compression results for ResNet-32 using fractional numbers as effective quantization bits, such as 0.4 and 1.2, that have not been available previously.
All layers, except the first and the last layers, are followed by FIeXOR components sharing the same M structure (thus, storage foorprint of M is ignorable). SGD optimizer is used with momentum of 0.9 and a weight decay factor of 10 10-5 Initial learning rate is 0.1, which is decayed by 0.5 at the th and th epoch. As learning rate decays, Stanh is empirically multiplied by 2 to cancel
5

---

Under review as a conference paper at ICLR 2020
Tost Accuracy (%)
XOR Training Method
Analog
STE
FleXOR
XOR
Forward
sign
Lanh
sign
FI-XR
Backward
Identity
((aanh)
(tann)
Analog XOR
STE
XOR
Binary
IR
Binary
Output
-1 or +1
+1)
-1 or +1
100
150
200
Epoch
Figure 5: Test accuracy comparison on ResNet-32 (for CIFAR-10) using various XOR training methods. Nout = 10, Nin 8, = (thus, 0.8bit/weight), and Stanh 10.
6k
5k
Test Accuracy (%)
Suss
85
Count
3k
Stark
21c
75-
Stark 50
1k
70-
0k:
0
50
100
150
200
-0.3
-0.2
-0.1
0.0
0.1
0.2
0.3
Epoch
Enerypted Weight Value
Figure 6: Test accuracy and distribution of encrypted weights (at the end of training) of ResNet-32 on CIFAR-10 using various Stanh and the same Nout, Nin, and g Figure
out the effects of weight decay on encrypted weights. Batch size 128 and scaling factors of a are randomly initialized between 0.01 and 0.19. is the number of bits to represent binary codes for quantization. We provide some useful training insights below with relevant experimental results.
1) Use small Ntap (such 2): FleXOR should be able to select the best out of 2.N possible outputs that are randomly selected from larger Noue search space. Encryption performance of XOR gates is determined by randomness of 2.V.n output candidates, and is enhanced by increasing Ham- ming distance that is basically achieved by large Nout- Now, let Ntop be the number of I's in a row of M Another method to enhance encryption performance to increase Ntop so as to increase the number of shuffles (through more XOR operations) using encrypted bits to generate quantized bits such that correlation between quantized bits is reduced. Large Ntap- however induces vanishing gradient problems in Eq. {5) or increased approximation error in Eq. (6). Hence, in practice, FleXOR training with small Neap converges well with high test accuracy. Studying a training algorithm to understand complex XOR-gate network with large Neap would be an interesting research topic that is beyond the scope of this work. Subsequently, we show experimental results using Neop 2 in the remainder of this paper.
2) Use 'tanh' rather than STE: Since forward propagation for XOR gate only needs a sign function, the STE method is also applicable to XOR-gate gradient calculations. Another alternative method to model an XOR gate is to use Eq. 3 for both forward and backward propagation as if the XOR is modeled in an analog manner (then, real number XOR outputs are quantized through STE). We compare three different XOR modeling schemes in Figure 5 with test accuracy measured when encrypted weights and XOR gates are converted to binary for inference. FleXOR training method shows the best result because sign function for forward propagation enables estimating the impact of binary XOR computations on the loss function and b) ((tan) for backward propagation approximates the Heaviside step function better compared STE. Note that limited gradients from the tanh function eliminate the need for weight clipping, which is often required for quantization- aware training schemes (Courbariaux al. 2015 Xu et al. 2018).
3) Optimize Stanh: Stanh controls the smoothness of the tanh function for near near-zero inputs. Large Stank employs large gradient for small inputs and, hence, results in well-clustered encrypted weight

---

Under review as a conference paper at ICLR 2020
92
93
Test Accuracy (%)
90
Tost. Accuracy (%)
92
88
91
86
90
(y=1, I, Noa=
841
19=2, 1o
(g=2, Nin Nout= 10)
82
(1=1, No. 8, Nout 10)
(g=1, No==20)
(g=2, No)=
80
88
100
200
300
400
500
0.5
1.0
1.5
2.0
Epoch
Number of Bits Weight
Figure 7: Test accuracy of ResNet-32 on CIFAR10 using learning rate warmup and various q. Nin. and Nout- The results the right side obtained by 5 runs.
Table 1:
Weight compression comparison of ResNet-20 and ResNet-32 on CIFAR-10. For FleXOR, we use warmup scheme, S1anh=10, and Now==20. All of quantization schemes in the table follow the form of binary codes with q=1, and hence, the amount of computations becomes the same. FleXOR, however, saves on/off-chip memory requirements further becasue N /Nout less than
ResNet-20
ResNet-32
FP
Compressed
Diff.
FP
Compressed
Diff.
BWN (1 bit)
92.68% F
87.44%
-5.24%
93.40% 93.40%
89.49%
-4.51 %
BinaryRelax (1 bit)
92.68%
87.82%
-4.86%
90.65%
-2.00%
LQ-Net (1 bit)
92. 1
90.10%
-1.90%
FleXOR (1.0 bit)
90.44%
-1.47%
91.36%
-0.97%
FleXOR (0.8 bit)
89.71%
-2.10%
90.78%
-1.55%
91.87%
92.33%
FleXOR (0.6 bit)
89.04%
-2.83%
90.30%
-2.03%
FleXOR (0.4 bit)
87.85%
-4.02%
89.61 %
-2.72%
values as shown Figure 6 Too large of a Stanh. however, hinders encrypted weights from being finely-tuned through training. For FleXOR, Stanh is a a syper-parameter to be optimized.
4) Learning rate and Stanh warmup: Learning rate starts from O and linearly inereases to reach the initial learning rate at a certain epoch as a warmup. Learning rate warmup is heuristic scheme, but being widely accepted to improve generalization capability mainly by avoiding large learning rate in the initial phase (He et al., 2018 Gotmare al. 2019). Similarly, Stanh starts from 5 to linearly increases to 10 using the same warmup schedule of the learning rate.
5) Try various q, Nin, and Nout: Using warmup scheme for 100 epochs and learning rate decay by 50% at the 350th, 400th and t50th epoch, Figure 7rusennt test accuracy of ResNet-32 with various g. Ne.. and Nout- For g>11, different M configurations are constructed and then shared across all layers. Note that even for the 0.4bit/weight configuration (using q=1, N and 2o), high accuracy close to 89% is achieved. 0.8bit/weight be achieved by two different configurations (as shown on the right side of Figure 7 using (g=1, N,,=8, -o)) or (q=2, N Nout=20 Interestingly, those two configurations show almost the same test accuracy, which implies that FleXOR is able to provide a linear relationship between the amount of encrypted weights and model accuracy (regardless internal configurations). In general, lowering g reduces the amount of computations with quantized weights and a high Nout value is necessary for very low number of bits per weight since Nr, also needs be high enough increase the Hamming distance.
We compare quantization results of ResNet-20 and ResNet-32 on CIFAR- 10 using different com- pression schemes in Table (with full free-iinnision activation). BWN (Rastegari al.] 2016), Bina- ryRelax Yin et al. 2018). and LQ-Net (Zhang al. 2018) propose different training algorithms
Reeooteed by BinaryRelax paper Yin et al. 2018
"Reported by I.Q-Net paper /7hag et al. 2018
'Based in
where replaced the ReL. functions with
leaky ReL.U functions.

---

Under review as a conference paper at ICLR 2020
Table 2:
Weight compression comparison of ResNet-18 on ImageNet using various compression schemes. All of quantization schemes in the table follow the form of binary codes with g=1, and hence, the amount of computations becomes the same, FleXOR, however, saves on/off-chip memory requirements further becasue / is less than 1.
Methods
Bits/ Weight
Top-1
Top-5
Storage Saving
Full Precision (He et al. 2016)
32
69.6%
89.2%
1x
BWN Rastegari et al. 2016)
60.8%
83.0%
32x
ABC-Net (Lin et al. 2017)
62.8%
84.4%
32x
BinaryRelax Yin et al. 2018)
63.2%
85.1%
~ 32x
0.8
63.8%
84.8%
~ 40
FleXOR (Noo.: 20)
0.6
62.0%
83.7%
53x
0.5
61.3%
83.1%
~ 64x
for the same quantization scheme (i.e., binary codes). The main idea of these methods is to mini- mize quantization eπrоr and to obtain gradients from full -ulliireciio weights while the loss function aware of quantization. Because all quantization schemes in Table Juses uses q=1 and binary codes, the amount of computations using quantized weights is the same. FleXOR, however, allows re- duced memory footprint and bandwidth which are critical to improve energy efficiency of inference engine designs (Han et al., 2016 Ahn al., 2019). Note that even though achieving the best ac- curacy for 1.Obit/eight is not the main purpose of FleXOR (e.g.. XOR gate may be redundant for Ni. oouc), FleXOR shows the minimum accuracy drop for ResNet-20 and ResNet-32 as shown in Table D probably because a) FleXOR enables model optimization integrated with quantization and encyption units instead of considering the local quantization error and b) FleXOR calculates gra- dients for encrypted weights by employing 'tanh' (instead of STE) that approximates a Heaviside step function.
5 EXPERIMENTAL RESULTS ON IMAGENET
In order to show that FleXOR principles can be extended to larger models, we choose ResNet- 18 on ImageNet (Russakovsky et al.) 2015). We use SGD optimizer with momentum of 0.9 and initial learning rate of O. Batch size is 128, weight decay factor is 10-5, and Stanh is 10. Learning rate reduced by half at the 70th 100th and 130th For warmup, during initial ten epochs, Stanh and learning rate increase linearly from 5 and 0.0, respectively, to initial values.
Table 2 shows the comparison on model accuracy of ResNet-18 when weights are compressed by quantization (and additional encryption by FleXOR) while activations maintain full precision. Train- ing the entire model for ImageNet including FleXOR components is successfully performed. In Table 2] BinaryRelax and BWN do not modify the underlying model architecture, while ABC-Net introduces a new block structure of the convolution for quantized network designs. FleXOR achieves the best top-1 accuracy even with only 0. O.8hit/weight and demonstrates improvement of model aceu- racy as the number of bits per weight increases. Refer to Appendix for the accuracy graph with q=1 and more results with q-2.
6 CONCLUSION
This paper proposes an encryption algoridhm/architectuue FleXOR, as framework to further com- press quantized weights. Eneryption designed to produce more outputs than inputs by increasing the Hamming distance of output functions when output functions are linear functions of inputs. Out- put functions are implemented as a combination XOR gates which are included in the model to find encrypted and quantized weights through gradient descent while using the tanh function for backward propagation. FleXOR is able to provide fractional numbers of bits for weights and, thus, much wider trade-offs between weight storage and model accuracy. Experimental results show that ResNet on CIFAR- 10 and ImageNet can be represented by sub 1-bit/weight compression with high accuracy.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Daehyun Ahn, Dongsoo Lee, Taesu Kim, and Jae-Joon Kim. Double Viterbi: Weight encoding for high compression ratio and fast on-chip reconstruction for deep neural network. In International Conference Learning Representations (ICLR), 2019.
Yoshua Bengio, Nicholas Lonard, and Aaron Courville, Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv:1308.3432, 2013.
Matthieu Courbariaux, Yoshua Bengio, and Jean-Pierre David. BinaryConnece Training deep neural networks with binary weights during propagations. In Advances in Neural Information Processing Systems, pp. 3 3123-3131, 2015.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations (ICLR), 2019.
Akhilesh Gotmare, Nitish Shirish Keskar, Caiming Xiong, and Richard Socher. A closer look deep learning heuristics: Learning rate restarts, warmup and distillation. In International Conference on Learning Representations (ICLR), 2019.
Song Han, Huizi Mao, and William J. Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and Huffman coding. International Conference on Learning Representations (ICLR), 2016.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. 2016 IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp. 770- 778, 2016.
Tong He, Zhi Zhang, Hang Zhang, Zhongyue Zhang, Junyuan Xie, and Mu Li. Bag of tricks train convolutional neural networks for image classification. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
Sangil Jung, Changyong Son, Seohyung Lee, Jinwoo Son, Jae-Joon Han, Youngjun Kwak, Sung Ju Hwang, and Changkyu Choi. Learning to quantize deep networks by optimizing quantization intervals with task loss. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4350-43359 2019.
J. Kahn, G. Kalai, and N. Linial. The influence of variables on boolean functions. In Proceedings of the 29th Annual Symposium on Foundations of Computer Science, SFCS '88, pp. 68-80, 1988.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
Fengfu Li and Bin Liu. Ternary weight networks. 1051v:160..04711, 2016.
Hao Li, Soham De, Zheng Xu, Christoph Studer, Hanan Samet, and Tom Goldstein. Training quantized nets: A deeper understanding. In Advances Neural Information Processing Systems, pp. 5813-5823, 2017.
Xiaofan Lin, Cong Zhao, and Wei Pan. Towards accurate binary convolutional neural network. In Advances in Neural Information Processing Systems, pp. 345-353, 2017.
Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. XNOR-Net: Imagenet
classification using binary convolutional neural networks. In ECCV, 2016.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), (1(3))211-255) 2015. doi: 10.1007/s1 1233-01-----1
Pierre Stock, Armand Joulin, Rmi Gribonval, Benjamin Graham, and Herv Jgou. And the bit goes down: Revisiting the quantization of neural networks. arXiv: 2019.

---

Under review as a conference paper at ICLR 2020
Kuan Wang, Zhijian Liu, Yujun Lin, Ji Lin, and Song Han. Haq: Hardware-awart automated quan- tization with mixed precision. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8612-8620, 2019.
Chen Xu, Jianqiang Yao, Zhouchen Lin, Wenwu Ou, Yuanbin Cao, Zhirong Wang, and Hongbin Zha. Alternating multi-bit quantization for recurrent neural networks. In International Conference on Learning Representations (ICLR), 2018
Penghang Yin, Shuai Zhang, Jiancheng Lyu, Stanley Osher, Yingyong Qi, and Jack Xin. Bina- ryrelax: A relaxation approach for training deep neural networks with quantized weights. SIAM 2018.
Journal on Imaging Sciences, 11(4):2205-22
Dongqing Zhang, Jiaolong Yang, Dongqiangzi Ye, and Gang Hua. Lq-nets: Learned quantization for highly accurate and compact deep neural networks. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 365-382, 2018.
Chenzhuo Zhu, Song Han, Huizi Mao, and William J. Dally. Trained ternary quantization. In International Conference on Learning Representations (ICLR), 2017.
10

---

Under review as a conference paper at ICLR 2020
A
APPENDIX
A.1 EXAMPLE OF A XOR-GATE NETWORK STRUCTURE REPRESENTATION
In Figure !
outputs of XOR-gate network are given
f/1 I1 2 Ty 2 J4
8/2 = I1 + 22
y3 I1 + 22 2 T3
91 = 23 + I4
y5 I2 +
y6 = 22 2 I3 2 IA-
Equivalently, the same structure above be represented in a matrix as
0
1
1
0
M +
0
1
1
0
1
1
Note that elements of M 3 are matched with coefficients of ys <i 6). For two vectors = {y1, V2, U3, V4. Us: yo and (a1, T2, 23, )4), the following equation holds:
M
where element-wise addition and multiplication are performed by 'XOR' and 'AND' function, re- spectively. In Eq. 47 Ntap (i.e., the number of '1's row) is
A.2 SUPPLEMENTARY DATA FOR BASIC FLEXOR TRAINING PRINCIPLES
A Boolean XOR gate can be modeled F (2)) (- 1) sign (21) sign(r2) if is replaced with as shown in Table 3
(1))(
sign (22)
[11, 22
Table 3: An XOR gate modeling using F (x1,222))
In Eq. 77. forward propagation for ya is expressed
Va (x 22, ma) = (-1)² sign(z)) sign(x2) (2gn(
(9)
while partial derivative of ya with respect to æn is given as (not derived from Eq. 9
Dys Om
Seanh (-1)
1aPP11ttnh?(aa tanh?(x) Stanh)) tanh(x22 Stanh) tanh Stanh),
(10)
or as
dy Oz
Seanh (-1)² tanh? (z) Stant)) sign (I2) sign(r3)-
(11)
We choose Eq. instead of Eq. 10 explained in Section 3.
As shown in Figure large Stanh yields sharp transitions for near-zero inputs. Such sharp approxi- mation of the Hcaviside step function produces large gradient values for small inputs and encourages encrypted weights to be separated into negative or positive values. Too large Smanh- however, has the same isseus of too large learning rate.
11

---

Under review as a conference paper at ICLR 2020
1.0
Stanh
14
Stant =1
Stanh =2
Stanh =2
12
Stanh
Stanh
0.5
Sumh 10
Stanh = 10
Sumh
Stank=
0.0
-0.5
2
-1.0
-2
-1
0
-2
-1
2
tanh(E Stant))
y = Seant (1 tanh2 Stann))
Figure 8: The left graph shows hyperbolic tangent (y= myy=tam tanh(x Stanh)) graphs with various scaling factors (Stanh). The right graph shows their derivatives. These graphs support the arguments of Optimize Stanh" in Section 4.
99.2
0.30
99.2
0.30
0.25
0.25
Tost Accuracy (%)
Test Accuracy (%)
98.5
98.5
0.20
Train TaoRs
Train LarS
98.1
0.15
98.1
0.15
Nin Now 11m==10
Ni==8, 8, Now Pom==2 20
U7.7
-oa=110
0.10
97.7
= 12, N Now
0.10
10
N
No Ni=116, Nour=20
97.4
0.05
97.4
0.05
97.0
0.00
97.0
0.00
5000
10000
15000
20000
5000
10000
15000
20000
Global Step
Global Step
Figure 9: Test accuracy and training loss of LeNet-5 on MNIST when number of '1's in each row of M is fixed to be (N 2). N Nout is 10 or 20 to generate, effectively, 0.4, 0.6, or 0.8 bit/weight quantization. With low Noop of M MNIST training presents less variations on training loss and test accuracy that in Figure
Figure presents training loss and test accuracy when Ntap = 2 and Nout is 10 or 20. Compared with Figure 3 NNop presents improved accuracy for the cases of high compression configura- tions (e.g., Nin = 4 and Nou = 10). We use Neep = 2 for CIFAR-10 and ImageNet, since low Ntap avoids gradient vanishing problems high approximation errors in Eq.
(q.
Figure 10 plots the distribution of encrypted weights at different training steps when each row of M randomly assigned with {0, 1} (i.e., Neap is Nt./2 on average) or assigned with only two I's (N) Due to gradient calculations based on tanh and high Stanh, encrypted weights tend be clustered on the the left or right (near-zero encrypted weights become less as Ntap increases) without weight clipping.
A.3 SUPPLEMENTARY EXPERIMENTAL RESULTS CIFAR- 10 AND IMAGENET
In this subsection, we additionally provide various graphs and accuracy tables for ResNet models on CIFARIO and ImageNet. We also present experimental results from wider hyper-parameters searches including 2 with two separate M configurations (with the same No, and Nout for two M matrices).
12

---

Under review as a conference paper at ICLR 2020
50k
30k
26k
step
26k
step
10x stop
10% step
20k
14k step
20k
14k
step
step
Count
Count
-0.050 -0.025 0.000
0.025
0.050
-0.050
-0.025
0.000
0.025
0.050
Encrypted Weight Value
Encrypted Weight Value
Figure 10: Distribution of encrypted weight values for FCI layer of LeNet-5 at different training steps using Stanh = 100 and Nout 10. (Left): M randomly filled (Noop Nnn/2). (Right): Ntap = 2 for every row of S
300
300
S
250
Staoh 5
250
8
Staah 100
Staah 1lah=10
Staah 550 50
Staah
S
S 100
Count
Count
150
150
100
100
50
-0.3
-0.2
-0.1
0.0
0.1
0.2
0.3
-0.3
-0.2
-0.1
0.0
0.1
0.2
0.3
Encrypted Weight Value
Encrypted Weight Value
First convolution layer Layerl
(b) Last convolution layer in Layerl
1500
Stant
1500
Stanh
Stant 5
Suan =5
1250
S 100
1250
Sundh 10
Stanh 50
Stanh 50
1000
Stant
1000
S
Count
Count
750
750
500
250
250
0
-0.3
-0.2
-0.1
0.0
0.1
0.2
0.3
-0.3
-0.2
-0.1
0.0
0.1
0.2
0.3
Encrypted Weight Value
Encrypted Weight Value
(c) First. convolution layer Layer2
(d) Last convolution layer in Layer2
Figure 11: Distributions of encrypted weights (at the end of training) in various layers of ResNet- 32 on CIFAR-10 using various Stanh and the same Nouts No,, and g as Figure(5] The ResNet-32 network mainly consists of three layers according to the feature map sizes: Layerl, Layer2 and Layer3.
13

---

Under review as a conference paper at ICLR 2020
90
87
Tist. Accurscy (%)
85
82
80
77
0.05
75
1r=0.1
1r=0.2
72
70
0
25
50
75
100
125
150
175
200
Epoch
(a) Initial Learning Rate (0.1): Test accuracy of ResNet-32 on CIFARIO using the learning schedule in Figure and various initial learning rates (0.05, 0.1, 0.2. 0.5).
90
no weight clipping
weight clipping
Test Accuracy (56)
80
70
0
25
50
75
100
125
150
175
200
Epoch
(b) No Weight Clipping: Test accuracy of ResNct-32 on CIFAR1O using the learning schedule in Figure 5 As for weight clipping, we restrict the encrypted weights to be ranged (-2.0/ Stah/ +2.0/ Stanh)) As can be observed, the red line implics that weight clipping is effective with FleXOR.
65
Test Accuracy (%, Top-1)
60
55
50
MM MMmMyn MMM
45
nu weight decay
our recipe with weight decay(10 $)
40
0
25
50
75
100
125
150
175
Epoch
(c) Weight Decay Factor (10 Two graphs depict accuracy of ResNet-18 ImageNet with without weight decay. learning rate
red line (no weight decay) reduced by half at the 100 130 and 1h epochs. The learning rate of the blue line (with weight decay) is reduced by half at 7 100 and 130 epochs. With weight decay (blue graph), despite slow convergence in the early training steps, model accuracy is eventually higher than the red without weight decay scheme.
Figure 12: Comparison of various hyper-parameter choices for CIFAR-10 or ImageNet.
14

---

Under review as a conference paper at ICLR 2020
92
90
Test Accuracy (%)
86
84
11=1, No. 8, Nout 20)
(g=1, Ni= Nou 20) 20)
78
(q=1, Nr.. 11, Now=
76
(q=1, No. 20, Nout (ou==2) 20)
100
200
300
400
500
Epoch
(a) Test accuracy using 4 1.
92
90
Test Accuracy (%)
86
84
82
(q=2, Nin =8, Nout 20)
(q=2, 1in=12, N Nout=20
78
(q=2, Ni. = 16, N 7mm.=20 20)
76
(q=2, Ni 20, Nous 20)
100
200
300
400
500
Epoch
(b) Test accuracy using q Compared the above plots (Figure 13ah. this figure shows that combination of multiple O for a binary code can lead stable learning curves and higher model accuracy.
Figure 13: Test accuracy of ResNet-32 CIFAR-10 using learning rate warmup (for 100 epochs)
and Noat = 20
15

---

Under review as a conference paper at ICLR 2020
Bits/Weight
ResNet-20
ResNer-32
Comp. Ratio
FP
32
91.87%
92.33%
1.0x
Nn= = 1::0 N Now==10
1.0
90.21%
-1.66%
91.40%
-0.93%
29.95
-9. 10
0.9
90.03%
-1.84%
91.28%
-1.05%
31.82
-in=8, Nou==10
0.8
89.59%
-2288%
90.86%
-1.47%
35.32
Nm=77, 10
0.7
89.24%
-2.63%
90.54%
-1.79%
39.68x
N =, Now==10
0.6
89.21%
-2.66%
90.41%
-1.92%
45.27
N 5, 1o
0.5
88.54%
-3.33%
89.82%
2.51%
52.70x
Table Weight compression comparison of ResNet-22 and ResNet-32 on CIFAR-10 10 when Nout = 10. Parameters and recipes not described in the table are the same in Table [ We also present compression ratio for fractional quantized ResNet-32 when one scaling factor (a) is assigned to each output channel.
ResNet-20
ResNet-32
Quant.
Diff.
Quant.
Diff.
TWN (ternary)
92.68%
88.65%
-4.03%
93.40%
90.94%
-2.46%
GinaryRelax (ternary)
92.68%
90.07%
-1.91%
93.40%
92.04%
-1.36%
TTQ (ternary)
91.77%
91.13%
-0.64%
92.33%
92.37%
+0.04%
LQ-Net (2 bit)
92.10%
91.80%
-0.30%
FleXOR(qq = 2, Nout
20)
22, 2.0 bit/weight
91.28%
-0.59%
91.99%
-0.34%
-1. 1.8 bit/weight
90.96%
-0.91%
92.01%
-0.32%
1,,, 1.6 bit/weight
91.87%
90.81% %
-1.06%
92.33%
91.55%
-0.78%
No.=14, 1.4 bit/weight
90.22%
-1.65%
91.33%
-1.00%
=,, 1.2 bit/weight
89.88%
1.99%
90.78%
-155%
FleXOR(g 2, Nous=
10)
N,.=10,, 2.0 bit/weight
91.06%
-0.81%
92.61%
+0.28%
=,, 1.8 bit/weight
91.00%
-0.87%
92.09%
-0.24%
Nn 1.6 bit/weight
91.87%
90.82%
-1.05%
92.33%
91.96%
-0.37%
Nn, 1.4 bit/weight
90.81%
-1.06%
91.63%
-0.70%
Nn.66, 1.2 bit/weight
90.28%
-1.59%
91.32%
-1.01%
Table 5: Weight compression comparison of ResNet-20 and ResNet-32 CIFAR-10 using learning
rate warmup (for 100 epochs) and 2. As mentioned in Figure 4 multiple M can be combined for multi-bit quantization schemes. Then, the number of scaling factors should be doubled. FleXOR with q 2 and two different M structures achieve full-precision accuracy when both N.. and Nout are 10.
65
Test Accuracy (%, Top-1)
(q=1, N..=16, Nm.==20
60
(q=1, N.. 12, Noue 20)
50
45
20
40
60
80
100
120
140
Epoch
Figure 14: Test accuracy (Top-1) of ResNet-18 on ImageNet using FIeXOR.
16

---

Under review as a conference paper at ICLR 2020
Methods
Biig/Weight
Top-1
Top-5
Full Precision (He et al. 2016)
32
69.6%
89.2%
TWN (Li & Liu| 2016)
ternary 2
61.8%
84.2%
ABC-Net (Lin et al.| 2017)
63.7%
85.2%
BinaryRelax Yin et al.] 2018)
ternary
66.5%
87.3%
TTQ(1.5x Wide) (Zhu al.
2017)
ternary 2
66.6%
87.2%
LQ-net (Zhang al. 2018)
68.0%
88.0%
QII (Jung et al. 2019)
68.1%
88.3%
1.6 (0.8x2)
66.2%
86.7%
FleXOR (q 2, Nout 20)
1.2 (0.6x2)
65.4%
86.0%
0.8 (0.4x2)
63.8%
85.0%
Table 6: Weight compression comparison of ResNet- 18 on ImageNet when 2. Since is 2, we also list the other compression schemes which use 2-bit or ternary quantization scheme for model compression.
17


Reviews:
[{"review_id": "ryxHNSeUtS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: \nThe authors propose quantize the weights of a neural network by enabling a fractional number of bits per weight. They use a network of differentiable XOR gates that maps encrypted weights to higher-dimensional decrypted weights to decode the parameters on-the-fly and learn both the encrypted weights and the scaling factors involved in the XOR networks by gradient descent.\n\nStrengths of the paper:\n- The method allows for a fractional number of bits per weights and relies of well-known differentiable approximations of the sign function. Indeed, virtually any number of bits/weights can be attained by varying the ratio N_in/N_out.\n- The papers displays good results on ImageNet for a ResNet-18.\n\nWeaknesses of the paper:\n- Some arguments that are presented could deserve a bit more precision. For instance, quantizing to a fractional number of bits per weights per layer is in itself interesting. However, if we were to quantize different layers of the same network with distinct integer  ratio of bits per weights (say 1 bit per weight for some particular layers and 2 bits per weight for the other layers), the average ratio would also be fractional (see for instance \"Hardware-aware Automated Quantization with Mixed Precision\", Wang et al., where the authors find the right (integer) number of bits/weights per layer using RL). Similarly, using vector quantization does allow for on-chip low memory: we do not need to re-instantiate the compressed layer but we can compute the forward in the compressed domain (by splitting the activations into similar block sizes and computing dot products). \n- More extensive and thorough experiments could improve the impact of the paper. For instance, authors could compress the widely used (and more challenging) ResNet-50 architecture, or try other tasks such as image detection (Mask R-CNN). The table is missing results from: \"Hardware Automated Quantization\", Wang et al ; \"Trained Ternary Quantization\", Zhu et al ; \"Deep Compression\",  Han et al; \"Ternary weight networks\", Li et al (not an extensive list).\n- Similarly, providing some code and numbers for inference time would greatly strengthen the paper and the possible usage of this method by the community. Indeed, I wonder what the overhead of decrypting the weights on-the-fly is (although it only involves XOR operations and products)\n- Small typos: for instance, two points at the very end of section 5.\n\nJustification fo rating:\nThe proposed method is well presented and illustrated. However, I think the paper would need either (1) more thorough experimental results (see comments above, points 2 and 3 of weaknesses) or (2) more justifications for its existence (see comments above, point 1 of weaknesses).", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "rkelDbynFr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposed a fractional quantization method for deep net weights. It adds XOR gates to produce quantized weight bits compared with existing quantization method. It used tanh functions instead of a straight-through estimator for backpropagation during training. With both designs, the proposed method outperformed the existing methods and offered sub bit quantization options.\n\nThe use of XOR gates to improve quantization seems novel. The sub bit quantization achieved by this method should be interesting to the industrial. It significantly improved the quantization rate with slightly quality degradation. With 1 bit quantization, it outperformed the state-of-the-art. The results seem thorough and convincing.\n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper."}, "extra_text": {}}, {"review_id": "HJenC0h6FH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a new approach for quantizing neural networks. It looks at binary codes for quantization instead of the usual lookup tables because quantization approaches that rely on lookup tables have the drawback that during inference, the network has to fetch the weights from the lookup table and work with the full precision values, which means that the computation cost is remains the same as the non-quantized network. The paper presents FleXOR gates, a fast and efficient logic gate for mapping bit sequences to binary weights. The benefit of the gate is that the bit sequence can be shorter than the binary weights which means that the code length can be less than 1 bit per weight.\n\nThe paper also proposes an algorithm for end-to-end training of the quantized neural networks. It proposes the use of tanh to approximate the non-differentiable Heaviside step function in the backward pass.\n\nNovelty\n\nThe idea of using logic gates for dequantization is interesting and (as far as I know) novel. One can imagine, that specialized hardware build on this idea could very efficient for inference (in terms of energy cost).\n\nWriting\n\nThe paper is very well written and completed with great visualizations and pseudocode. Kudos to the authors, I really enjoyed reading it. However, I do not think it is justified to go over the 8 page soft limit. I would recommend that the authors perhaps shorten section 3 or remove figure 9 to fit it into 8 pages.\n\nSignificance/Impact\n\nThe paper is motivated by the high computation cost of working with full precision values. But this paper also works with full precision weights, since it has a full precision scaling factor (alpha) and, as far as I understood, works with full precision values during forward propagation. This means that there likely are no computational savings when compared to lookup tables.\n\nThe evaluation section lacks experiments that evaluate the computational savings. The baselines should include quantization methods based on lookup tables, and there should be a comparison of computational costs. The baselines that are presented (BWN etc.) offer a tradeoff between accuracy and computational costs, yet they are only compared in accuracy. I would strongly recommend including the computational cost of each method in the evaluation section.\n\nOverall assessment:\nWhile I enjoyed reading this paper, I am leaning towards rejection due to the shortcomings of the evaluation section.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work studies parameter quantization using binary codes and proposes an encryption algorithm/architecture to compress quantized weights and achieve fractional numbers of bits per weight, and to perform decryption using XOR gates. The authors conduct experiments on datasets including ImageNet to evaluate their scheme.
Much of the concern from reviewers relates to baseline comparison and details around that. Specifically, R1 believes that the submission could have a bigger impact if authors could conduct more thorough experiments, e.g. compressing more widely-used and challenging architecture of ResNet-50, or trying tasks such as image detection (Mask R-CNN). The authors' responded to that and mentioned their choice of the current experimental setting is to facilitate comparison with previous works (baselines), which use similar experimental settings. Nevertheless, the baseline methods could have been attempted by the authors on broader tasks, or more widely-used architectures could have been investigated by authors on the baseline methods. As a result, R1 was not convinced. To ensure the paper receives the attention it deserves, I recommend considering a more thorough evaluation of the proposed method against baseline methods.
Decision: Reject