# Layer-Wise Compressive Training for Convolutional Neural Networks
## 1 Introduction  
Convolutional Neural Networks (CNNs) represent a class of machine learning algorithms that are able to extrapolate complex data representations from unstructured data, e.g., images, text, and audio. Starting from the astounding results obtained by [PERSON] et al. [1] during the 2012 ImageNet competition, CNN models have been improved substantially, achieving classification accuracies that go beyond those of a human being. Today's CNNs are considered the state-of-the-art in several application domains, such as medical diagnosis [2; 3; 4], personal home assistants [5], surveillance systems [6], and self-driving vehicles [7]. The most important factor behind the rise of CNNs, and deep learning in general, is the availability of efficient computing platforms able to speed-up the training stages, e.g., GPUs. Nonetheless, the adoption of CNNs in pervasive applications, such as those of the Internet-of-Things, is still far from being achieved. The reason is simple: the requirements of storage capacity and computational resources do not fit into low-power embedded architectures. To better grasp these concepts, let us consider the internal structure of a CNN. Figure 1 depicts the feature maps of the well-known VGG-16 model. It consists of several layers, each with a proper function: (_i_) the _input layer_ (red box), which produces a first transformation of the input image to be fed to the next convolutional layers; (_ii_) the _output layers_ (gray boxes), which are perceptron-based artificial neural networks in charge of producing the final answer on the classification task; and (_iii_) the _hidden layers_, where the features extraction takes place. Figure 1 also shows the dimension of each layer; by summing up the contributions of all the layers, it results that even a small CNN model like the VGG-16 requires 138 MB of memory storage, and that is for weights alone. One should consider the feed-forward phase of a CNN asks for additional memory to store the partial results produced by eachlayer. It is therefore clear that CNNs have to be squeezed in order to fit into architectures with few MB of memory available. Several strategies have been proposed to shrink pre-trained CNN models. Knowledge distillation [8; 9] and low rank approximation [10; 11] are representative examples of _non-destructive_ techniques. The original model serves as a reference point to generate a child model with a smaller structure and unchanged accuracy, or to generate linear-separable kernels that can be represented with a concise mathematical representation. By contrast, _destructive_ methods do apply a direct modification of the original CNN structure. Under this category, weights pruning [12; 13; 14] and weights quantization [15; 16] are the most widely studied techniques. Inspired by the mechanisms that control and modify neural synapses in the human brain, the pruning approach is based on the idea of removing unimportant neural connections by setting their weights to zero. Quantization, instead, is used to reduce the number of bits per weight, hence the memory footprint, and thus the precision of each weight representation. Extreme quantization techniques include the possibility to quantize weights to binary [17; 18] or ternary [19; 20] numbers, and thus to further reduce the complexity of multiply-and-accumulate operations (replaced with simple shift operations). However, binary and ternary networks are proved to lead to huge accuracy degradations when evaluated on large datasets [21]. This is because most solutions apply a blind quantization, without taking into account how weights evolve during the training stages. In order to fill this gap, [PERSON]. [22] introduced an efficient \(-\sigma\)_compressive training_ algorithm that accounts for weights represented in the constrained space \((-\sigma,0,\sigma)\). Differently from other state-of-the-art strategies, the compressive training searches for optimal \(\sigma\)'s such that accuracy degradation is minimized.