# Layer-Wise Compressive Training for Convolutional Neural Networks
## 4 A Greedy Approach for Compressive Training
### Optimization  
**Step 3--re-Training** The retraining phase is applied in order to recover the accuracy loss due to pruning. The retraining is applied after pruning at first, and then after each optimization loop.  
**Step 4--Compression** It is the compressive training described in Section 3. The weights are projected in a sub-dimensional space composed by just three values for each layer, i.e., \((-\sigma,0,+\sigma)\), with \(\sigma\) defined layer-wise.  
**Step 5--Validation** The model is validated in order to quantify the accuracy loss due to compression, and thus to decide if it is worth continuing with further compressions. Validation is a paramount step for the greedy approach as it actually enables an accuracy-driven optimization. The accuracy \(Acc_{n}\) is evaluated and stored after each compression epoch \(n\).  
**Step 6--Condition 1 (C1)** The accuracy recorded during the \(n\)-th epoch (parameter \(Acc_{n}\)) is used to determine if the CNN model can be further compressed, as in Equation (18). The accuracy of the pre-trained model (\(Acc_{0}\)) works as baseline, whereas the parameter \(\epsilon\) represents the user-defined accuracy loss tolerance:  
\[Acc_{n}>Acc_{0}-\epsilon. \tag{18}\]  
It is worth noticing that the higher the \(\epsilon\), the higher the compression of the CNN model. The framework takes a larger execution time for small values of \(\epsilon\). This is due to the increased complexity in selecting a good combination of layers that allows matching the user's constraints.  
Figure 5: AlexNet on Imagenet, layers after the sorting algorithm; the sparsity value \(S\) is reported for each layer.  
C1 can lead to two possible branches. If Equation (18) holds true, then the algorithm goes to step 7; otherwise, the quit condition C2 is evaluated.  
**Step 7--Update** This stage is applied _if_ Equation (18) is verified. The counter \(N\) indicates how many layers of the sorted list can be compressed. Each and every time C1 is evaluated as true, and \(N\) is incremented by \(\Delta N\). The latter represents another granularity knob, hence on the speed of the framework; \(\Delta N\) is mainly defined by the network size: the larger the CNN model, the larger the \(\Delta N\).  
**Step 8--Condition 2 (C2)** This last condition is based on the maximum number of epochs \(n_{max}\), a user-defined hyperparameter. At the \(n\)-th iteration, if more than \(n_{max}\) epochs are elapsed, the algorithm stops, else the flow iterates over step 3.  
Figure 6 shows how accuracy evolves during the optimization loop: we report the validation's trend during the algorithm's epochs (blue-line), also reporting the accuracy values after both the re-Training and Compression phases (red-line). To better understand the behavior of the model during its compression, we recall that, for each loop, the algorithm first runs the retraining phase (based on a float32 back-propagation algorithm) and then a feature projection of the full precision model in the constrained solution space (this induces some accuracy drop). As the plot suggests, the performance loss is recovered within each retraining phase. The peak of loss reflects the addition of a new layer in the compressible subset list. There are layers that influence more the performance drop, but in general, after some epochs, the network reconstructs the information lost.