# Layer-Wise Compressive Training for Convolutional Neural Networks
## 5 Results
### Performance  
For the validation of the proposed technique, we consider the trade-off between the accuracy loss and the compression ratio. The two performance metrics are described in Equations (19) and (20). The former represents the accuracy loss and is defined as the difference in terms of accuracy percentage between the original full-precision model (\(Model_{FP}\)) and the compressed one (\(Model_{C}\)). The latter describes the compression rate defined as the ratio between the memory storage needed to save the original model's parameters and the storage needed for saving the compressed model after the encoding. On each compressed layer, we apply the weights encoding illustrated in [22], using a 4-bit counter. In the original model, all the weights' parameters (\(N_{0}\)) have to be saved in 32-bit; for the compressed model, only the different \(N^{\sigma}\) values (one per each compressed layer) and the total number of weights of the full-precision layers \(N^{FP}\) need 32 bit precision. \[AccuracyLoss\quad[\%]\quad=\quad Accuracy(Model_{FP})-Accuracy(Model_{C}), \tag{19}\]  
\[CompressionRate\quad[\times]\quad=\quad\frac{N^{0}\cdot 32_{bit}}{N^{FP} \cdot 32_{bit}+Length(Enc)\cdot 1_{bit}+N^{\sigma}\cdot 32_{bit}}. \tag{20}\]  
We first focus on the performance on CIFAR10 and CIFAR100 datasets with \(AlexNet\), \(VGG19_{bn}\), and \(ResNet110\). Tables 1 and 2 summarize the obtained results. In both tables, the first row reports the name of the CNN model with the baseline accuracy in parentheses; the second row reports the experimental results in terms of Top-1 accuracy and accuracy loss; the last row describes the compression rate after weights encoding for each different CNN model. Obtained numbers refer to a user-defined accuracy loss of <1%. The numbers suggest not only that the accuracy constraint is successfully met, but they also indicate that very large compression rates (e.g., 26.4\(\times\) for the AlexNet model) are easily achieved. This proves the adopted rationale is sound and also applicable to very complex CNN model; it allows to preserve useful information just removing redundant, or less significant, parameters on the less significant layers. \begin{table}
\begin{tabular}{l c c c c c c} \hline \multirow{2}{*}{**CIFAR10**} & \multicolumn{2}{c}{**AlexNet (77.22\%)**} & \multicolumn{2}{c}{**VGG19\({}_{bn}\) (93.02\%)**} & \multicolumn{2}{c}{**ResNet110 (93.81\%)**} \\ \cline{2-7}  & **Accuracy** & **Acc.Loss** & **Accuracy** & **Acc.Loss** & **Accuracy** & **Acc.Loss** \\ \hline OUR & 76.44\% & 0.78\% & 92.20\% & 0.82\% & 93.32\% & 0.49\% \\ \hline Compression Rate & \multicolumn{2}{c}{26.4\(\times\)} & \multicolumn{2}{c}{6.7\(\times\)} & \multicolumn{2}{c}{8.0\(\times\)} \\ \hline \end{tabular}
\end{table} 
Table 1: Validation accuracy results of compressive training on CNNs trained on CIFAR10. For each CNN model, the accuracy loss (\(Acc.Loss\)) is referred to the baseline accuracy reported in parentheses. To further understand our approach, we detail two network structures before and after the greedy compressive training. Table 3 shows the ResNet20 architecture on CIFAR10: the two first columns show the layers ID and their input size; the next columns show the intra-layer sparsity percentage and the bit-width adopted to represent the weights, both for the full precision model (_FP_) and the compressed model (_CM_). The last rows report the sparsity of the resulting net, the compression rate w.r.t. the original model, and the final accuracy. Table 4 shows the same kind of metrics for AlexNet trained on CIFAR100. Since our framework applies automatic decisions on the number of layers to compress, for each CNN and each dataset adopted, the results may change substantially, also depending on the accuracy constraint and the number of iterations run.