Quantization Analysis for Face Image Detection Through Dense Neural Networks

The range of applications of facial recognition has greatly expanded, especially in critical fields as crime prevention, bank account access and payments. Machine learning has greatly helped the development and the diffusion of this biometric measurement, making it easier to develop and to implement. The increasing popularity in mobile systems introduces the problem of lowpower networks with high accuracies. This requirement is in contrast to the current trend of more complex and deeper networks. Many methods have been proposed to implement less computationally and memory intensive systems with low accuracy deterioration: knowledge distillation and quantization are some examples. This thesis focuses on post-training quantization on deep structures: two versions of a DenseNet architecture have been considered, both descended from a more complex network through knowledge distillation. The investigation concentrates on two main families of quantization techniques: scalar quantization and vector quantization. Three scalar quantization techniques have been taken into account. Firstly a reduced bit-width version of Floating-Point has been considered to cut the required memory with an almost null degradation on the accuracy. To further reduce the computational complexity, Dynamic Fixed-Point and Power of 2 Weights representations have been adopted. Regarding vector quantization, K-Means Clustering has been considered. This method is able to reduce the storage requirement. All the networks have been implemented in Caffe. Scalar quantization has been applied thanks to Ristretto framework, an expansion of Caffe, while KMeans Clustering have been carried out in Python thanks to the sklearn library. The quantization has been performed taking into account a layer-by-layer approach. Firstly Convolutional and Fully Connected layers have been approximated. Then the attention moves on Batch Normalization layers, which are more sensitive to quantization and involves highly computational consuming operations. To solve this latter issue, Folding has been applied to Batch Normalization layers to avoid complex operation, e.g. division and square root, with null accuracy degradation. Each version of the two reference DenseNet networks have been classified in terms of accuracy through the LFW test, a standard procedure in Neural Network. To identify the solutions with an acceptable accuracy degradation, quantized DenseNets have been compared to the reference 32-bit FP networks. Good results in terms of both accuracy and model compression have been achieved with reduced Floating-Point, K-Means Clustering and an hybrid approach between the two: – 8-bit Floating-Point is characterized by an almost null accuracy degradation (< 0.2) and a compression factor of the network size equal to 4 for both the networks; K-Means Clustering reaches good results if each layer is associated to a different set of 16 clusters. Both the networks achieve a parameter compression factor higher then 7.9, limiting the accuracy degradation to 2.48 in DenseNet 1.0 and to 4.35 for 2.0 version. An hybrid approach between the ones described above restricts accuracy degradation to 2.45 and 4.45, while the parameter size are decreased by a factor 8 and the total network size by a factor 5.3. Power of 2 Weights representation unfortunately introduces too high accuracy deterioration in all the scenarios. Adopting Dynamic Fixed-Point good results can be achieved if only Convolutional and Fully Connected layers are quantized. Nevertheless, when Batch Normalization is taken into account, the representation format fails to cover the dynamic range of the new parameters.