# Image-Captioning Model Compression
## 4 Experiments
### Decoder Pruning  
Results of pruning using different methods and pruning coefficients are presented in Figures 1 and 2. Blue and orange lines on both figures are correspondent to \(l_{1}\) pruning, while red and green lines are correspondent to random pruning. \"True\" represents pruning the embedding layer and \"False\" represents not doing so. NNZ means \"number of non-zero parameters\", which is the common measure for comparing pruning techniques. As it can be seen, random pruning is strictly worse than \(l_{1}\) pruning. This could be due to the fact that diminishing less valuable weights of the model expectedly influences the model's quality metrics less than removing only random parameters of the model. The other fact is that although not pruning the embeddings layer helps to maintain greater quality for big pruning coefficients for the Up-Down model, it almost doesn't show any difference for the AoANet model. Additionally, the great decline in metrics could be observed for the Up-Down model starting from \(\alpha=0.1\), and for AoANet starting from \(\alpha=0.5\). Taking this into account, in any case we are not interested in the area of Up-Down model pruning where the choice of pruning or not pruning embeddings could play any role. As said before, the biggest pruning coefficients (which lead to a bigger model compression) with which models still show good quality are 0.1 and 0.5 for Up-Down and AoANet, respectively. Increasing these values could lead to a drop in metrics. More detailed comparisons of model results near this drop, along with the unpruned model quality, are reported in Table 3. The table proves the observations obtained from figures. The chosen boundary values for pruning coefficients help to reduce model size by 10% and losing only 0.6 CIDEr points for the Up-Down model, and reducing model size by 50% and losing only 1.5 CIDEr points for the AoANet model. Such a great reduction in AoANet could indicate that the overall model is bigger and has more parameters than needed to show comparable results with its architecture. Furthermore, this model's weights could be quite sparse, and lots of them could be close to 0. In this case, such extensive pruning wouldn't lead to big loss in performance. \begin{table}
\begin{tabular}{l c c c c c c c c} \hline \hline
**Decoder Scale Factor** & **B@1** & **B@4** & **M** & **R** & **C** & **S** & **Size** & **\#Params** \\ \hline \multicolumn{10}{c}{Up-Down} \\ \(\gamma=1\) & 79.4 & 35.8 & 27.3 & 56.7 & 119.5 & 21.1 & 193 MB & 50.1 M \\ \(\gamma=\textbf{0.5}\) & **79.4** & **35.2** & **27.1** & **56.6** & **119.2** & **20.9** & **68.8 MB** & **18 M** \\ \(\gamma=0.25\) & 78.3 & 33.2 & 26.2 & 55.5 & 114.3 & 19.8 & 27.5 MB & 7.2 M \\ \hline \multicolumn{10}{c}{AoANet} \\ \(\gamma=1\) & 79.8 & 38.3 & 29.6 & 58.3 & 129.2 & 22.1 & 323.4 MB & 84.8 M \\ \(\gamma=\textbf{0.5}\) & **79.5** & **38.5** & **29.5** & **58.3** & **129.1** & **22.1** & **100.7 MB** & **26.4 M** \\ \(\gamma=0.25\) & 78.5 & 36.4 & 28.6 & 57.4 & 122.9 & 20.9 & 35.2 MB & 9.2 M \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Evaluation results of all of tested models decoders reduced with different scale factors. Both “Size” and “#Params” columns refer to decoder. Figure 1: CIDEr and SPICE validation scores dependence on pruning coefficient \(\alpha\) for different pruning methods applied to the Up-Down model. “L1” indicates that \(l_{1}\) pruning was used, whilst “Random” indicates that random pruning was used. “True” indicates pruning of the embedding layer and “False” indicates no pruning.