TASK DEFINITION: In this task, you are given the abstract of a research paper. Your task is to generate a summary of this abstract. Your summary should not be very short, but it's better if it's not more than 30 words.
PROBLEM: Whereas it is believed that techniques such as Adam, batch normalization and, more recently, SeLU nonlinearities ``solve'' the exploding gradient problem, we show that this is not the case and that in a range of popular MLP architectures, exploding gradients exist and that they limit the depth to which networks can be effectively trained, both in theory and in practice. We explain why exploding gradients occur and highlight the {\it collapsing domain problem}, which can arise in architectures that avoid exploding gradients. 

 ResNets have significantly lower gradients and thus can circumvent the exploding gradient problem, enabling the effective training of much deeper networks, which we show is a consequence of a surprising mathematical property. By noticing that {\it any neural network is a residual network}, we devise the {\it residual trick}, which reveals that introducing skip connections simplifies the network mathematically, and that this simplicity may be the major cause for their success.

SOLUTION: We show that in contras to popular wisdom, the exploding gradient problem has not been solved and that it limits the depth to which MLPs can be effectively trained. We show why gradients explode and how ResNet handles them.

PROBLEM: Quantization of a neural network has an inherent problem called accumulated quantization error, which is the key obstacle towards ultra-low precision, e.g., 2- or 3-bit precision. To resolve this problem, we propose precision highway, which forms an end-to-end high-precision information flow while performing the ultra-low-precision computation. First, we describe how the precision highway reduce the accumulated quantization error in both convolutional and recurrent neural networks. We also provide the quantitative analysis of the benefit of precision highway and evaluate the overhead on the state-of-the-art hardware accelerator. In the experiments, our proposed method outperforms the best existing quantization methods while offering 3-bit weight/activation quantization with no accuracy loss and 2-bit quantization with a 2.45 % top-1 accuracy loss in ResNet-50. We also report that the proposed method significantly outperforms the existing method in the 2-bit quantization of an LSTM for language modeling.

SOLUTION: precision highway; a generalized concept of high-precision information flow for sub 4-bit quantization 

PROBLEM: Recent work has shown that performing inference with fast, very-low-bitwidth
 (e.g., 1 to 2 bits) representations of values in models can yield surprisingly accurate
 results. However, although 2-bit approximated networks have been shown to
 be quite accurate, 1 bit approximations, which are twice as fast, have restrictively
 low accuracy. We propose a method to train models whose weights are a mixture
 of bitwidths, that allows us to more finely tune the accuracy/speed trade-off. We
 present the “middle-out” criterion for determining the bitwidth for each value, and
 show how to integrate it into training models with a desired mixture of bitwidths.
 We evaluate several architectures and binarization techniques on the ImageNet
 dataset. We show that our heterogeneous bitwidth approximation achieves superlinear
 scaling of accuracy with bitwidth. Using an average of only 1.4 bits, we are
 able to outperform state-of-the-art 2-bit architectures.

SOLUTION:
We introduce fractional bitwidth approximation and show it has significant advantages.