# Exploring the Potential for Real-Time Vision Transformer-Level Precision on Face Recognition Scenarios through Binarization on Embedded Systems

Luis S. Luevano, Miguel González-Mendoza, Yoanna Martínez-Díaz, Heydi Méndez-Vázquez

► To cite this version:

Luis S. Luevano, Miguel González-Mendoza, Yoanna Martínez-Díaz, Heydi Méndez-Vázquez. Exploring the Potential for Real-Time Vision Transformer-Level Precision on Face Recognition Scenarios through Binarization on Embedded Systems. ICCVW 2023 - IEEE/CVF International Conference on Computer Vision Workshops, Oct 2023, Paris, France. 2023. hal-04393662

HAL Id: hal-04393662

https://inria.hal.science/hal-04393662v1

Submitted on 14 Jan 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Exploring the Potential for Real-Time Vision Transformer-Level Precision on Face Recognition Scenarios through Binarization on Embedded Systems

<figure><img src="image_2.png" /></figure>

Luis S. Luevano¹, Miguel González-Mendoza²,
Yoanna Martínez-Díaz³, Heydi Méndez-Vázquez³

¹ Inria, Rennes, France

² Tecnológico de Monterrey, Nuevo León, Mexico

³ Advanced Technologies Application Center (CENATAV), Havana, Cuba

<figure><img src="image_3.png" /></figure>

## Introduction

* **Context:** Efficient Face Recognition (FR) is a key challenge in Computer Vision, prompting the exploration of strategies that balance accuracy and computational resources. In this work, we conduct a thorough comparative analysis of three modern approaches for FR: Lightweight Convolutional Neural Networks (CNNs), Vision Transformers (ViTs), and Binary Neural Networks (BNNs).

**Motivation:** To enhance efficiency for real-time scenarios on embedded devices, we investigate BNNs, utilizing binary weights and quantization techniques to reduce memory and computation requirements. We also explore Lightweight CNN and ViT-based approaches on FR scenarios, with the potential for global context comprehension using self-attention mechanisms for improving accuracy.

**Goal:** To suggest concrete refinements that could be present in BNN designs for harnessing the recent progress of these three modern avenues. We specifically focus on efficient and accurate FR scenarios being available on extremely hardware-constrained platforms, such as the Nvidia Jetson Nano.

## On Quantization and Binarization

* **State of the art Quantization:** Quantization analyses in the state of the art are limited to 16 and 8-bit computations for FR applications.

* **Challenges on Binarization:** Extreme quantization, also known as Binarization, using 1-bit neural networks, offers potential efficiency benefits but faces challenges including information loss, computation stability, and achieving real-time performance on affordable hardware.

* **Recent efforts using Binarization on FR:** Newer methods, such as BinaryFaceNet, show a remarkable efficiency performance with less than 13% of verification accuracy penalty on the AGEDB-30 benchmark against the state of the art. This design is the only one enabling face verification in real-time (6.25 FPS) on a single ARM core of an Nvidia Jetson Nano using the Larq Compute Engine framework.

## Vision Transformer Efficiency vs Accuracy on ImageNet

<figure><img src="image_4.png" /><figcaption>Figure: Modern ViT and Lightweight CNN designs benchmarked on ImageNet against inference time latency, taken from the SwiftFormer paper. We note that SwiftFormer reports the best accuracy-to-efficiency trade-off, showing potential for FR applications.</figcaption></figure>

<figure><img src="image_5.png" /><figcaption>Figure: Efficient self-attention designs, taken from the SwiftFormer paper. (a) ViT Self-Attention, (b) EdgeFace Transposed Self-Attention, (c) MobileViT V2 Separable Self-attention, and (d) SwiftFormer's Efficient Additive Attention.</figcaption></figure>

## FR-specific methods with Lightweight CNNs & ViTs

<table><thead><tr><th>Method</th><th>Type</th><th>FLOPs</th><th>Params</th><th>LFW</th><th>CFP-FP</th><th>AGEDB-30</th></tr></thead><tbody><tr><td>VarGFaceNet</td><td>L.CNN</td><td>1.02G</td><td>4.9M</td><td>99.73%</td><td>97.67%</td><td>97.5%</td></tr><tr><td>MobileFaceNet-1.5</td><td>L.CNN</td><td>933.3M</td><td>2.0M</td><td>97.33%</td><td><strong>99.71%</strong></td><td>97.56%</td></tr><tr><td>ShuffleFaceNet-1.5</td><td>L.CNN</td><td>577.5M</td><td>2.6M</td><td>97.38%</td><td>97.25%</td><td>97.31%</td></tr><tr><td>ShuffleFaceNet-2.0</td><td>L.CNN</td><td>1.05G</td><td>4.5M</td><td>99.62%</td><td>97.56%</td><td>97.28%</td></tr><tr><td>GhostFaceNetV2-1</td><td>L.CNN</td><td>272.25M</td><td>6.88M</td><td><strong>99.86%</strong></td><td>99.33%</td><td><strong>98.62%</strong></td></tr><tr><td>EdgeFace¹</td><td>Hybrid</td><td>153.99M</td><td>1.77M</td><td>99.68%</td><td>94.46%</td><td>95.72%</td></tr><tr><td>MobileFaceFormer²</td><td>Hybrid</td><td><strong>130.08M</strong></td><td><strong>1.38M</strong></td><td>99.60%</td><td>96.79%</td><td>97.69%</td></tr></tbody></table>

Table: Lightweight CNN and efficient Hybrid method performance on popular face verification benchmarks, trained on the MS1M-V3 dataset.

¹EdgeFace was trained on WebFace-260M.

² MobileFaceFormer was trained on CASIA-WebFace, FLOPs estimated as 2×MAdds reported on the original paper.

## BNN Accuracy on FR and Efficiency Evaluations

<table><thead><tr><th>Method</th><th>1-bit<br>MACs</th><th>32-bit<br>MACs</th><th>Mem.<br>(MB)</th><th>Params</th><th>Time<br>(img/sec)</th><th>FPS</th><th>LFW</th><th>CFP-FP</th><th>AGEDB-30</th></tr></thead><tbody><tr><td>BinaryDenseNet45</td><td>1.59G</td><td>59.7M</td><td>4.2</td><td>13.1M</td><td>6.2s</td><td>0.16</td><td><strong>99.28%</strong></td><td><strong>92.88%</strong></td><td><strong>91.03%</strong></td></tr><tr><td>BinaryDenseNet37</td><td>1.12G</td><td>48.9M</td><td>2.6</td><td>8.0M</td><td>4.4s</td><td>0.22</td><td>99.17%</td><td>92.59%</td><td>90.72%</td></tr><tr><td>BinaryDenseNet28</td><td>893.2M</td><td>49.99M</td><td>1.8</td><td>4.5M</td><td>3.7s</td><td>0.27</td><td>99.17%</td><td>92.11%</td><td>90.72%</td></tr><tr><td>QuickNet</td><td>431.7M</td><td>6.8M</td><td>2.21</td><td>12.7M</td><td>1.8s</td><td>0.55</td><td>98.97%</td><td>92.00%</td><td>89.00%</td></tr><tr><td>QuicNet-small</td><td>258.3M</td><td><strong>3.4M</strong></td><td>2.0</td><td>12.1M</td><td>1.1s</td><td>0.86</td><td>98.55%</td><td>90.65%</td><td>88.00%</td></tr><tr><td>BinaryFaceNet</td><td><strong>184.9M</strong></td><td>3.8M</td><td><strong>1.3</strong></td><td><strong>506K</strong></td><td><strong>0.16s</strong></td><td><strong>6.25</strong></td><td>95.07%</td><td>77.93%</td><td>75.12%</td></tr></tbody></table>

Table: Efficiency and face verification accuracy results tested on popular face recognition benchmarks. The inference runtime efficiency was tested on a single ARM core of an Nvidia Jetson Nano. All BNNs were trained on the MS1M-V3 dataset.

## Research directions

* **Further ViT-based architecture analysis on FR:** Newer ViT-based methods, such as SwiftFormer, have the potential for improving results on FR benchmarks, based on their reported results on ImageNet performance, while also enhancing efficiency.

* **Self-attention mechanisms on BNNs:** the efficiency improvements present in separate and additive attention present an opportunity for implementing such mechanisms in BNN blocks and potentially coupling them with quantization capabilities.

* **Knowledge Distillation:** this training methodology could drastically improve FR performance across different scenarios by using a Transformer architecture as teacher and a design such as BinaryFaceNet as student, for highly-efficient and accurate FR.

* **Privacy-preservation analysis on Quantized approaches for FR:** with more focus on privacy-preserving computations for mitigating gradient leakage and protecting user privacy, binarization has proved to be a useful tool in general-purpose scenarios but further privacy analyses can be performed and are paramount for the adoption of mainstream AI technology for FR scenarios.

## More on Face Recognition through Binarization

<figure><img src="image_6.png" /></figure>

<figure><img src="image_7.png" /></figure>

<figure><img src="image_8.png" /></figure>

<figure><img src="image_9.png" /></figure>

## Acknowledgements

This work was partially funded by the SOTERIA H2020 project. SOTERIA received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement No101018342. This content reflects only the author's view. The European Agency is not responsible for any use that may be made of the information it contains.

The authors would like to thank the financial support from Tecnologico de Monterrey through the "Challenge-Based Research Funding Program 2022". Project ID # E120- EIC-GI06 - B-T3 - D