# Layer-Wise Compressive Training for Convolutional Neural Networks
## 2 Background  
### Overview of Training Algorithms  
The term _training_ refers to the process used to find (sub-)optimal values for all weights and biases in a neural network. The most popular training procedures make use of iterative methods that follow the derivatives of a loss function to reach a minimum cost. The loss function, here denoted with \(\mathcal{L}\), is the error between the output of the network and the expected one. Given \(\vec{\omega}\) as the tensor that  
Figure 1: Topology of the VGG-16 model. represents all the weights in a CNN model, \(\mathcal{L}\) is defined as in Equation (1), with \(\vec{x}\) the input of the CNN:  
\[z=\mathcal{L}(\vec{\omega},\vec{x}). \tag{1}\]  
Let us consider the \(k\)-th component in the input space \(\vec{x}_{k}\). The associated loss function \(z_{k}\) is the difference between \(\vec{\xi}_{k}\), i.e., the predicted class associated to \(\vec{x}_{k}\), and the correct result \(\vec{\delta}_{k}\), as reported in Equation (2):  
\[\mathcal{L}(\vec{\omega},\vec{x})=\|\vec{\delta}_{k}-\vec{\xi}_{k}\|^{2}. \tag{2}\]  
In other words, the loss function can be seen as a measure of similarity between the outputs \(\vec{\xi}_{k}\) and \(\vec{\delta}_{k}\). The same formulation can be extended to a batch of input tensors \(\mathcal{B}\) by defining the loss as the sum of all the single errors on the elements of the batch itself; this is true as errors are always positive. Hence, minimizing the sum implies minimizing each contribution. To prevent this summation to diverge, the loss function is averaged by the number of elements in the batch \(\mathcal{N}\) as reported in Equation (3):  
\[\mathcal{L}(\vec{\omega})=\frac{1}{\mathcal{N}}\sum_{\mathcal{B}}\|\vec{ \delta}_{k}-\vec{\xi}_{k}\|^{2}. \tag{3}\]  
The stochastic gradient descent (SGD) is an iterative optimization method that can serve to minimize a loss function. It may reach the optimality of the solution when the objective function is convex and it does not stop in a _saddle point_, namely a point where derivatives are zero. However, when applied to non-convex functions, SGD allows reaching a local minimum. In the context of deep learning models, local minima represent a reasonable solution and a good compromise: CNNs are too complex to be handled with more sophisticated methods, whereas SGD is simple and effective. As the name suggests, the SGD implements a stochastic exploration of the solution space which is aimed to reach the optimal \(\vec{\omega}\). The search evolves following the negative slope of the objective function. More specifically, the SGD exploits the definition of gradient, that is the vector of the first-order derivatives of the objective function as described by Equation (4). When calculated in a generic  
To overcome this issue, a practical solution is to use mini-batches. Instead of computing the sum over the whole \(\mathcal{B}\), each iteration just takes a sub-set of \(\mathcal{M}\) random samples as described in Equation (8), where \(\mathcal{B}_{\mathcal{M}}\) is called mini-batch. In such case, the gradient is approximated on the \(\mathcal{M}\) contributions:  
\[\vec{\
abla}\mathcal{L}\mathcal{M}(\vec{\omega}_{t})=\frac{1}{\mathcal{M}} \sum_{\mathcal{B}_{\mathcal{M}}}\vec{\
abla}\mathcal{L}(\vec{\omega}_{t},\vec {x}_{t}) \tag{8}\]  
such that the update formula becomes as in Equation (9):  
\[\vec{\omega}_{t+1}=\vec{\omega}_{t}-\eta\vec{\
abla}\mathcal{L}_{\mathcal{M}} (\vec{\omega}_{t}). \tag{9}\]  
Each iteration works with different mini-batches until every sample of the original batch \(\mathcal{B}\) is visited at least once. When exhausted, the mini-batches are newly generated using different random samples from \(\mathcal{B}\). This shuffling prevents any collateral biasing effect. Another property of mini-batched SGD is the ability to escape from a local minima or plateau \(\vec{\
abla}\mathcal{L}(\vec{\omega}_{t})=\vec{0}\).