# Layer-Wise Compressive Training for Convolutional Neural Networks
## 5 Results
### Comparison with the State-of-the-Art  
The analysis includes some of the most popular works on aggressive CNN compression: _Xnor-Net_[17], by [PERSON] et al., where both filters and feature maps are optimized in a binary space; _Ternary Weights Network_ (TWN) [20] where [PERSON] and [PERSON] et al. overcame the binary solution space adding the zero value as third quantized instance; _Trained Ternary Quantization_ (TTQ) [19], where [PERSON] et al. propose a new ternary quantization procedure able to use just 2-bit weights (with relative scaling factors) during CNN inference; _DoReFa-Net_[29], where [PERSON] el al. explored hybrid CNNs with different quantization widths for weights, gradients, and activations type with binary weights and 32-bit activations; we focus on the _1-32-32 DoReFa-Net_ in particular. For all the comparisons, the baseline is the accuracy obtained with full-precision models found in the PyTorch repository. In the following text, we use the accuracy loss as the main metric for comparison. Indeed, the results reported in the previous works do implement any encoding scheme and comparing the compressive rates might result in being quite unfair.  
Let us first consider the results on the CIFAR10 dataset. The first row in Table 5 describes the ResNet20 and ResNet56 baseline accuracies; each column reports the obtained results. For the first CNN model, our technique is able to outperform TTN's solution with just 0.55% of accuracy loss, whereas, for the ResNet56, the solution is closer to the baseline. However, for both networks, we set up the accuracy tolerance \(\epsilon\) at 1%, reaching a considerable compression rate (6.1\(\times\) and 6.9\(\times\)).  
Table 6 reports the experimental results obtained with the Imagenet dataset. We can see that, for AlexNet, the best solution is that achieved with TTN. In fact, they claim to be able to improve over full precision model. Their solution consists of a ternary weights CNN model (with relative scaling factors), except for the first and last layers, which are kept on float32 precision. Our solution outperforms all the other solutions, e.g., a 12.35% delta w.r.t. XNOR-Net. On the other hand, with the ResNet18 architecture, our model is able to outperform all considered state-of-the-art techniques, reaching just 1.86% of accuracy loss. For this set of experiments, considering the 1k-labels dataset complexity, we fixed the accuracy tolerance \(\epsilon\) at 2%, reaching compression rates of 10.9\(\times\) for AlexNet and 3.6\(\times\) for ResNet18.  
\begin{table}
\begin{tabular}{l c c c c c} \hline \hline  & & \multicolumn{2}{c}{**FP**} & \multicolumn{2}{c}{**CM**} \\ \cline{3-5}
**Layer** & **Shape** & \begin{tabular}{c} **Sparsity** \\ **[\%]** \\ \end{tabular} & \begin{tabular}{c} **Width** \\ **[Bit]** \\ \end{tabular} & \
\begin{tabular}{c} **Sparsity** \\ **[\%]** \\ \end{tabular} &
\begin{tabular}{c} **Width** \\ **[Bit]** \\ \end{tabular} \\ \hline Conv1 & (64,3,11,11) & 0 & 32 & 39 & 32 \\ Conv2 & (192,64,5,5) & 0 & 32 & 57 & 2 \\ Conv3 & (384,192,3,3) & 0 & 32 & 68 & 2 \\ Conv4 & (256,384,3,3) & 0 & 32 & 55 & 2 \\ Conv5 & (256,256,3,3) & 0 & 32 & 66 & 2 \\ Fc1 & (10,256) & 0 & 32 & 18 & 32 \\ \hline Final Sparsity & \multicolumn{2}{c}{0.00\%} & \multicolumn{2}{c}{60.61\%} \\ \hline Compression Rate & \multicolumn{2}{c}{-\(\times\)} & \multicolumn{2}{c}{26.7\(\times\)} \\ \hline Accuracy & \multicolumn{2}{c}{44.01\%} & \multicolumn{2}{c}{43.47\%} \\ \hline \hline \end{tabular}
\end{table}
Table 4: Layer-wise sparsity and width before (Full Precision) and after the compressive greedy technique (Compressed Model) application on AlexNet architecture (CIFAR100). Considering that convolutional layers have input shape \((n,c_{in},k_{h},k_{w})\), and fully-connected layers have input shape \((n,c_{in})\), where \(k_{h}\) and \(k_{w}\) are, respectively, the height and width of input planes in pixels, while \(c_{in}\) denotes the number of input channels and \(n\) the batch size.