# Layer-Wise Compressive Training for Convolutional Neural Networks
## 3 Compressive Training
### \(\sigma\)-Constrained Stochastic Gradient Descent  
During the initial stage, each layer \(l\) is assigned to a preliminary \(\sigma\) that is the statistical mean of the weights distribution \(W_{l}\). The mean is calculated as reported in Equation (11), where \(\psi\) is the number of weights after pruning, \(\omega_{i}\) are the weights, and \(\Delta\omega_{i}\) is the update resulting from the back-propagation of the error function:  
\[\sigma=\frac{1}{\psi}\sum_{i=0}^{\psi-1}|\omega_{i}+\Delta\omega_{i}|. \tag{11}\]  
In Equation (11), positive and negative contributions can be split as shown in Equation (12):  
\[\sigma =\frac{1}{\psi}\left(\sum_{\omega_{i}>0}\omega_{i}+\Delta\omega_{i }-\sum_{\omega_{i}<0}\omega_{i}+\Delta\omega_{i}\right) \tag{12}\] \[=\frac{1}{\psi}\sum_{i=0}^{\psi-1}|\omega_{i}|+\frac{1}{\psi} \left(\sum_{\omega_{i}>0}\Delta\omega_{i}-\sum_{\omega_{i}<0}\Delta\omega_{i} \right).\]  
Following the description provided in Section 2.1, where \(\Delta\vec{\omega}=-\eta\vec{\
abla}\mathcal{L}(\vec{\omega}_{k})\), \(\Delta\omega_{i}\) is defined as in Equation (13):  
\[\Delta\omega_{i}=-\eta\frac{\partial\mathcal{L}}{\partial\omega_{i}}(\vec{ \omega}_{k}). \tag{13}\]  
At the \((n+1)\)-th iteration, Equation (12) can be written as \(\sigma_{(n+1)}=\sigma_{n}+\Delta\sigma\), with \(\Delta\sigma\) as reported in Equation (14):  
\[\sigma_{n}=\frac{1}{\psi}\sum_{i=0}^{\psi-1}|\omega_{i}|;\Delta\sigma=\frac{1}{ \psi}\left(\sum_{\omega_{i}>0}\Delta\omega_{i}-\sum_{\omega_{i}<0}\Delta\omega _{i}\right). \tag{14}\]  
Each \(\sigma\) is updated with the weighted arithmetic mean of the gradients' components. Hence, if a generic weight is far from its optimal value, its partial derivative would strongly affect the value of \(\sigma\). While using the arithmetic mean for the first step represents a reasonable starting point, fine-tuning \(\sigma\) in a constrained solution space allows for carrying out an optimum search strategy. Figure 2: Weights distribution of the second layer of AlexNet trained on CIFAR10. After the preliminary training (**left**), after pruning (**center**), after compression (**right**). The constrained space can be seen as a semi-bisector described as in Equation (15), where \(\vec{s}\) is a vector having components in the form \(s_{j}=sign(\omega_{j})\) and norm \(\|s\|=\sqrt{\psi}\):  
\[\dot{e}=\frac{\vec{s}}{\|s\|}. \tag{15}\]  
By expressing the non-pruned weights in their vectorized form \(\vec{\omega}\), all their possible values, and thus the optimal solution, can be found along the tensor \(\vec{\omega}=\sigma_{n}\vec{s}\). However, some components of the direction could theoretically change because \(\omega_{j}\) might flip its sign. Therefore, the solution space is the set of all possible semi-bisectors, and the complete definition of \(s_{j}\) takes the form reported in Equation (16):  
\[s_{j}=sign(\omega_{j}+\Delta\omega_{j}). \tag{16}\]  
Let us assume an original \(5\times 5\) matrix pruned until only two positive weights \(\omega_{x}\), \(\omega_{y}\) and a single negative weight \(\omega_{z}\) are left. The solution space can be represented in three dimensions and becomes a semi-straight line, whose direction is \(\vec{s}=(1,1,-1)^{T}\). Figure 3 gives a pictorial representation of the solution space. If the actual solution \(\vec{\omega}=(\omega_{x},\omega_{y},\omega_{z})^{T}=(\sigma_{n},\sigma_{n},- \sigma_{n})^{T}\) belongs to the attractive valley of a minimum point \(H\), then \(-\vec{\
abla}\mathcal{L}(\vec{\omega})\) will point towards it. However, \(\vec{\omega}\) can only move along \(\vec{s}\), i.e., \(\dot{e}\). That means it is possible to find the projection of \(-\eta\vec{\
abla}\mathcal{L}(\vec{\omega})\) on \(\dot{e}\) to know which is the optimal \(\Delta\sigma\) as to approach as close as possible to \(H\). This is the ultimate sense of this algorithm: lowering or increasing \(\sigma\) to approach the minimum point in the best _approximated_ way.