user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Learning Recurrent Binary/Ternary Weights

Paper content:
Recurrent neural networks (RNNs) have shown excellent performance in processing sequence data.
However, they are both complex and memory intensive due to their recursive nature.
These limitations make RNNs difficult to embed on mobile devices requiring real-time processes with limited hardware resources.
To address the above issues, we introduce a method that can learn binary and ternary weights during the training phase to facilitate hardware implementations of RNNs.
As a result, using this approach replaces all multiply-accumulate operations by simple accumulations, bringing significant benefits to custom hardware in terms of silicon area and power consumption.
On the software side, we evaluate the performance (in terms of accuracy) of our method using long short-term memories (LSTMs) and gated recurrent units (GRUs) on various sequential models including sequence classification and language modeling.
We demonstrate that our method achieves competitive results on the aforementioned tasks while using binary/ternary weights during the runtime.
On the hardware side, we present custom hardware for accelerating the recurrent computations of LSTMs with binary/ternary weights.
Ultimately, we show that LSTMs with binary/ternary weights can achieve up to 12x memory saving and 10x inference speedup compared to the full-precision hardware implementation design.
Convolutional neural networks (CNNs) have surpassed human-level accuracy in various complex tasks by obtaining a hierarchical representation with increasing levels of abstraction BID3 ; BID31 ).
As a result, they have been adopted in many applications for learning hierarchical representation of spatial data.
CNNs are constructed by stacking multiple convolutional layers often followed by fully-connected layers BID30 ).
While the vast majority of network parameters (i.e. weights) are usually found in fully-connected layers, the computational complexity of CNNs is dominated by the multiply-accumulate operations required by convolutional layers BID46 ).
Recurrent neural networks (RNNs), on the other hand, have shown remarkable success in modeling temporal data BID36 ; BID12 ; BID6 ; ; ).
Similar to CNNs, RNNs are typically over-parameterized since they build on high-dimensional input/output/state vectors and suffer from high computational complexity due to their recursive nature BID45 ; BID14 ).
As a result, the aforementioned limitations make the deployment of CNNs and RNNs difficult on mobile devices that require real-time inference processes with limited hardware resources.Several techniques have been introduced in literature to address the above issues.
In BID40 ; BID22 ; BID29 ; BID42 ), it was shown that the weight matrix can be approximated using a lower rank matrix.
In BID34 ; BID14 ; ; BID1 ), it was shown that a significant number of parameters in DNNs are noncontributory and can be pruned without any performance degradation in the final accuracy performance.
Finally, quantization approaches were introduced in ; BID33 ; BID9 ; BID26 ; BID20 ; BID39 ; BID19 ; BID49 ; ; ) to reduce the required bitwidth of weights/activations.
In this way, power-hungry multiply-accumulate operations are replaced by simple accumulations while also reducing the number of memory accesses to the off-chip memory.Considering the improvement factor of each of the above approaches in terms of energy and power reductions, quantization has proven to be the most beneficial for hardware implementations.
However, all of the aforementioned quantization approaches focused on optimizing CNNs or fully-connected networks only.
As a result, despite the remarkable success of RNNs in processing sequential data, RNNs have received the least attention for hardware implementations, when compared to CNNs and fully-connected networks.
In fact, the recursive nature of RNNs makes their quantization difficult.
In BID18 ), for example, it was shown that the well-known BinaryConnect technique fails to binarize the parameters of RNNs due to the exploding gradient problem ).
As a result, a binarized RNN was introduced in BID18 ), with promising results on simple tasks and datasets.
However it does not generalize well on tasks requiring large inputs/outputs BID45 ).
In BID45 ; BID20 ), multi-bit quantized RNNs were introduced.
These works managed to match their accuracy performance with their full-precision counterparts while using up to 4 bits for data representations.In this paper, we propose a method that learns recurrent binary and ternary weights in RNNs during the training phase and eliminates the need for full-precision multiplications during the inference time.
In this way, all weights are constrained to {+1, −1} or {+1, 0, −1} in binary or ternary representations, respectively.
Using the proposed approach, RNNs with binary and ternary weights can achieve the performance accuracy of their full-precision counterparts.
In summary, this paper makes the following contributions:• We introduce a method for learning recurrent binary and ternary weights during both forward and backward propagation phases, reducing both the computation time and memory footprint required to store the extracted weights during the inference.•
We perform a set of experiments on various sequential tasks, such as sequence classification, language modeling, and reading comprehension. We
then demonstrate that our binary/ternary models can achieve near state-of-the-art results with greatly reduced computational complexity.
In this section, we evaluate the performance of the proposed LSTMs with binary/ternary weights on different temporal tasks to show the generality of our method.
We defer hyperparameters and tasks details for each dataset to Appendix C due to the limited space.
As discussed in Section 4, the training models ignoring the quantization loss fail to quantize the weights in LSTM while they perform well on CNNs and fully-connected networks.
To address this problem, we proposed the use of batch normalization during the quantization process.
To justify the importance of such a decision, we have performed different experiments over a wide range of temporal tasks and compared the accuracy performance of our binarization/ternarization method with binaryconnect as a method that ignores the quantization loss.
The experimental results showed that binaryconnect method fails to learn binary/ternary weights.
On the other hand, our method not only learns recurrent binary/ternary weights but also outperforms all the existing quantization methods in literature.
It is also worth mentioning that the models trained with our method achieve a comparable accuracy performance w.r.t. their full-precision counterpart.Figure 1(a
) shows a histogram of the binary/ternary weights of the LSTM layer used for characterlevel language modeling task on the Penn Treebank corpus. In
fact, our model learns to use binary or ternary weights by steering the weights into the deterministic values of -1, 0 or 1. Despite
the CNNs or fully-connected networks trained with binary/ternary weights that can use either real-valued or binary/ternary weights, the proposed LSTMs trained with binary/ternary can only perform the inference computations with binary/ternary weights. Moreover
, the distribution of the weights is dominated by non-zero values for the model with ternary weights.To show the effect of the probabilistic quantization on the prediction accuracy of temporal tasks, we adopted the ternarized network trained for the character-level language modeling tasks on the Penn Treebank corpus (see Section 5.1). We measured
the prediction accuracy of this network on the test set over 10000 samples and reported the distribution of the prediction accuracy in FIG0 . FIG0 (b) shows
that
the variance imposed by the stochastic ternarization on the prediction accuracy is very small and can be ignored. It is worth mentioning
that we also have observed a similar behavior for other temporal tasks used in this paper. FIG1 illustrates the learning
curves and generalization of our method to longer sequences on the validation set of the Penn Treebank corpus. In fact, the proposed training
algorithm also tries to retains the main features of using batch normalization, i.e., fast convergence and good generalization over long sequences. FIG1 (a) shows that our model
converges
faster than the full-precision LSTM for the first few epochs. After a certain point, the convergence
rate of our method decreases, that prevents the model from early overfitting. FIG1 (b) also shows that our training
method
generalizes well over longer sequences than those seen during training. Similar to the full-precision baseline,
our binary/ternary models learn to focus only on information relevant to the generation of the next target character. In fact, the prediction accuracy of our
models improves as the sequence length increases since longer sequences provides more information from past for generation of the next target character.While we have only applied our binarization/ternarization method on LSTMs, our method can be used to binarize/ternarize other recurrent architectures such as GRUs. To show the versatility of our method,
we repeat the character-level language modeling task performed in Section 5.1 using GRUs on the Penn Treebank, War & Peace and Linux Kernel corpora. We also adopted the same network configurations
and settings used in Section 5.1 for each of the aforementioned corpora. TAB4 6 summarizes the performance of our binarized/ternarized
models. The simulation results show that our method can successfully
binarize/ternarize the recurrent weights of GRUs.As a final note, we have investigated the effect of using different batch sizes on the prediction accuracy of our binarized/ternarized models. To this end, we trained an LSTM of size 1000 over a sequence
length of 100 and different batch sizes to perform the character-level language modeling task on the Penn Treebank corpus. Batch normalization cannot be used for the batch size of 1 as
the output vector will be all zeros. Moreover, using a small batch size leads to a high variance when
estimating the statistics of the unnormalized vector, and consequently a lower prediction accuracy than the baseline model without bath normalization, as shown in Figure 3 . On the other hand, the prediction accuracy of our binarization/ternarization
models improves as the batch size increases, while the prediction accuracy of the baseline model decreases. Figure 3 : Effect of different batch sizes on the prediction accuracy of the
character-level language modeling task on the Penn Treebank corpus. The introduced binarized/ternarized recurrent models can be exploited by various
dataflows such as DaDianNao (Chen et al. (2014) ) and TPU (Jouppi et al. (2017) ). In order to evaluate the effectiveness of LSTMs with recurrent binary/ternary weights
, we build our binary/ternary architecture over DaDianNao as a baseline which has proven to be the most efficient dataflow for DNNs with sigmoid/tanh functions. In fact, DaDianNao achieves a speedup of 656× and reduces the energy by 184× over a
GPU (Chen et al. (2014) ). Moreover, some hardware techniques can be adopted on top of DaDianNao to further speed
up the computations. For instance, showed that ineffectual computations of zero-valued weights can be skipped
to improve the run-time performance of DaDianNao. In DaDianNao, a DRAM is used to store all the weights/activations and provide the required
memory bandwidth for each multiply-accumulate (MAC) unit. For evaluation purposes, we consider two different application-specific integrated circuit
(ASIC) architectures implementing Eq. (2): low-power implementation and high-speed inference engine. We build these two architectures based on the aforementioned dataflow. For the low-power implementation
, we use 100 MAC units. We also use a 12-bit fixed-point representation
for both weights and activations of the full-precision
model as a baseline architecture. As a result, 12-bit multipliers are required to perform the recurrent computations. Note that using the
12-bit fixed-point representation for weights and activations guarantees no prediction
accuracy loss in the full-precision models. For the LSTMs with recurrent binary/ternary weights, a 12-bit fixed-point representation is only used for
activations and multipliers in the MAC units are replaced with low-cost multiplexers. Similarly, using 12-bit fixed-point representation for activations guarantees no prediction accuracy loss
in the introduced binary/ternary models. We implemented our low-power inference engine for both the full-precision and binary/ternary-precision models
in TSMC 65-nm CMOS technology. The synthesis results excluding the implementation cost of the DRAM are summarized in TAB7 . They show that using
recurrent binary/ternary weights results in up to 9× lower power and 10.6× lower silicon area
compared to the baseline when performing the inference computations at 400 MHz.For the high-speed design, we consider the same silicon area and power consumption for both the fullprecision and binary/ternary-precision models. Since the MAC units of the binary/ternary-precision model require less silicon area and power consumption as a result
of using multiplexers instead of multipliers, we can instantiate up to 10× more MAC units, resulting in up to 10× speedup compared to the full-precision model (see TAB7 ). It is also worth noting that the models using recurrent binary/ternary weights also require up to 12× less memory bandwidth
than the full-precision models. More details on the proposed architecture are provided in Appendix D.
In this paper, we introduced a method that learns recurrent binary/ternary weights and eliminates most of the full-precision multiplications of the recurrent computations during the inference.
We showed that the proposed training method generalizes well over long sequences and across a wide range of temporal tasks such as word/character language modeling and pixel by pixel classification tasks.
We also showed that learning recurrent binary/ternary weights brings a major benefit to custom hardware implementations by replacing full-precision multipliers with hardware-friendly multiplexers and reducing the memory bandwidth.
For this purpose, we introduced two ASIC implementations: low-power and high-throughput implementations.
The former architecture can save up to 9× power consumption and the latter speeds up the recurrent computations by a factor of 10.
Figure 4: Probability density of states/gates for the BinaryConnect LSTM compared to its fullprecision counterpart on the Penn Treebank character-level modeling task.
Both models were trained for 50 epochs.
The vertical axis denotes the time steps.
Figure 4 shows the probability density of the gates and hidden states of the BinaryConnect LSTM and its full-precision counterpart both trained with 1000 units and a sequence length of 100 on Penn Treebank corpus BID35 for 50 epochs.
The probability density curves show that the gates in the binarized LSTM fail to control the flow of information.
More specifically, the input gate i and the output gate o tend to let all information through, the gate g tends to block all information, and the forget gate f cannot decide to let which information through.
p and values centered around 1 for the input gate i.
In fact, the binarization process changes the probability density of the gates and hidden states during the training phase.

assistant: text: We propose high-performance LSTMs with binary/ternary weights, that can greatly reduce implementation complexity