Package: facenet-pytorch
Face Recognition Using PytorchPython3.103.93.8StatusThis is a repository for Inception Resnet (V1) models in pytorch, pretrained on VGGFace2 and CASIA-Webface.Pytorch model weights were initialized using parameters ported from David Sandberg'stensorflow facenet repo.Also included in this repo is an efficient pytorch implementation of MTCNN for face detection prior to inference. These models are also pretrained. To our knowledge, this is the fastest MTCNN implementation available.Table of contentsTable of contentsQuick startPretrained modelsExample notebooksComplete detection and recognition pipelineFace tracking in video streamsFinetuning pretrained models with new dataGuide to MTCNN in facenet-pytorchPerformance comparison of face detection packagesThe FastMTCNN algorithmRunning with dockerUse this repo in your own git projectConversion of parameters from Tensorflow to PytorchReferencesQuick startInstall:# With pip:pipinstallfacenet-pytorch# or clone this repo, removing the '-' to allow python imports:gitclonehttps://github.com/timesler/facenet-pytorch.gitfacenet_pytorch# or use a docker container (see https://github.com/timesler/docker-jupyter-dl-gpu):dockerrun-it--rmtimesler/jupyter-dl-gpupipinstallfacenet-pytorch&&ipythonIn python, import facenet-pytorch and instantiate models:fromfacenet_pytorchimportMTCNN,InceptionResnetV1# If required, create a face detection pipeline using MTCNN:mtcnn=MTCNN(image_size=<image_size>,margin=<margin>)# Create an inception resnet (in eval mode):resnet=InceptionResnetV1(pretrained='vggface2').eval()Process an image:fromPILimportImageimg=Image.open(<imagepath>)# Get cropped and prewhitened image tensorimg_cropped=mtcnn(img,save_path=<optionalsavepath>)# Calculate embedding (unsqueeze to add batch dimension)img_embedding=resnet(img_cropped.unsqueeze(0))# Or, if using for VGGFace2 classificationresnet.classify=Trueimg_probs=resnet(img_cropped.unsqueeze(0))Seehelp(MTCNN)andhelp(InceptionResnetV1)for usage and implementation details.Pretrained modelsSee:models/inception_resnet_v1.pyThe following models have been ported to pytorch (with links to download pytorch state_dict's):Model nameLFW accuracy (as listedhere)Training dataset20180408-102900(111MB)0.9905CASIA-Webface20180402-114759(107MB)0.9965VGGFace2There is no need to manually download the pretrained state_dict's; they are downloaded automatically on model instantiation and cached for future use in the torch cache. To use an Inception Resnet (V1) model for facial recognition/identification in pytorch, use:fromfacenet_pytorchimportInceptionResnetV1# For a model pretrained on VGGFace2model=InceptionResnetV1(pretrained='vggface2').eval()# For a model pretrained on CASIA-Webfacemodel=InceptionResnetV1(pretrained='casia-webface').eval()# For an untrained model with 100 classesmodel=InceptionResnetV1(num_classes=100).eval()# For an untrained 1001-class classifiermodel=InceptionResnetV1(classify=True,num_classes=1001).eval()Both pretrained models were trained on 160x160 px images, so will perform best if applied to images resized to this shape. For best results, images should also be cropped to the face using MTCNN (see below).By default, the above models will return 512-dimensional embeddings of images. To enable classification instead, either passclassify=Trueto the model constructor, or you can set the object attribute afterwards withmodel.classify = True. For VGGFace2, the pretrained model will output logit vectors of length 8631, and for CASIA-Webface logit vectors of length 10575.Example notebooksComplete detection and recognition pipelineFace recognition can be easily applied to raw images by first detecting faces using MTCNN before calculating embedding or probabilities using an Inception Resnet model. The example code atexamples/infer.ipynbprovides a complete example pipeline utilizing datasets, dataloaders, and optional GPU processing.Face tracking in video streamsMTCNN can be used to build a face tracking system (using theMTCNN.detect()method). A full face tracking example can be found atexamples/face_tracking.ipynb.Finetuning pretrained models with new dataIn most situations, the best way to implement face recognition is to use the pretrained models directly, with either a clustering algorithm or a simple distance metrics to determine the identity of a face. However, if finetuning is required (i.e., if you want to select identity based on the model's output logits), an example can be found atexamples/finetune.ipynb.Guide to MTCNN in facenet-pytorchThis guide demonstrates the functionality of the MTCNN module. Topics covered are:Basic usageImage normalizationFace marginsMultiple faces in a single imageBatched detectionBounding boxes and facial landmarksSaving face datasetsSee thenotebook on kaggle.Performance comparison of face detection packagesThis notebook demonstrates the use of three face detection packages:facenet-pytorchmtcnndlibEach package is tested for its speed in detecting the faces in a set of 300 images (all frames from one video), with GPU support enabled. Performance is based on Kaggle's P100 notebook kernel. Results are summarized below.PackageFPS (1080x1920)FPS (720x1280)FPS (540x960)facenet-pytorch12.9720.3225.50facenet-pytorch (non-batched)9.7514.8119.68dlib3.808.3914.53mtcnn3.045.708.23See thenotebook on kaggle.The FastMTCNN algorithmThis algorithm demonstrates how to achieve extremely efficient face detection specifically in videos, by taking advantage of similarities between adjacent frames.See thenotebook on kaggle.Running with dockerThe package and any of the example notebooks can be run with docker (or nvidia-docker) using:dockerrun--rm-p8888:8888-v./facenet-pytorch:/home/jovyantimesler/jupyter-dl-gpu\-v<pathtodata>:/home/jovyan/datapipinstallfacenet-pytorch&&jupyterlabNavigate to the examples/ directory and run any of the ipython notebooks.Seetimesler/jupyter-dl-gpufor docker container details.Use this repo in your own git projectTo use this code in your own git repo, I recommend first adding this repo as a submodule. Note that the dash ('-') in the repo name should be removed when cloning as a submodule as it will break python when importing:git submodule add https://github.com/timesler/facenet-pytorch.git facenet_pytorchAlternatively, the code can be installed as a package using pip:pip install facenet-pytorchConversion of parameters from Tensorflow to PytorchSee:models/utils/tensorflow2pytorch.pyNote that this functionality is not needed to use the models in this repo, which depend only on the saved pytorchstate_dict's.Following instantiation of the pytorch model, each layer's weights were loaded from equivalent layers in the pretrained tensorflow models fromdavidsandberg/facenet.The equivalence of the outputs from the original tensorflow models and the pytorch-ported models have been tested and are identical:>>> compare_model_outputs(mdl, sess, torch.randn(5, 160, 160, 3).detach())Passing test data through TF model

tensor([[-0.0142,  0.0615,  0.0057,  ...,  0.0497,  0.0375, -0.0838],
        [-0.0139,  0.0611,  0.0054,  ...,  0.0472,  0.0343, -0.0850],
        [-0.0238,  0.0619,  0.0124,  ...,  0.0598,  0.0334, -0.0852],
        [-0.0089,  0.0548,  0.0032,  ...,  0.0506,  0.0337, -0.0881],
        [-0.0173,  0.0630, -0.0042,  ...,  0.0487,  0.0295, -0.0791]])

Passing test data through PT model

tensor([[-0.0142,  0.0615,  0.0057,  ...,  0.0497,  0.0375, -0.0838],
        [-0.0139,  0.0611,  0.0054,  ...,  0.0472,  0.0343, -0.0850],
        [-0.0238,  0.0619,  0.0124,  ...,  0.0598,  0.0334, -0.0852],
        [-0.0089,  0.0548,  0.0032,  ...,  0.0506,  0.0337, -0.0881],
        [-0.0173,  0.0630, -0.0042,  ...,  0.0487,  0.0295, -0.0791]],
       grad_fn=<DivBackward0>)

Distance 1.2874517096861382e-06In order to re-run the conversion of tensorflow parameters into the pytorch model, ensure you clone this repowith submodules, as the davidsandberg/facenet repo is included as a submodule and parts of it are required for the conversion.ReferencesDavid Sandberg's facenet repo:https://github.com/davidsandberg/facenetF. Schroff, D. Kalenichenko, J. Philbin.FaceNet: A Unified Embedding for Face Recognition and Clustering, arXiv:1503.03832, 2015.PDFQ. Cao, L. Shen, W. Xie, O. M. Parkhi, A. Zisserman.VGGFace2: A dataset for recognising face across pose and age, International Conference on Automatic Face and Gesture Recognition, 2018.PDFD. Yi, Z. Lei, S. Liao and S. Z. Li.CASIAWebface: Learning Face Representation from Scratch, arXiv:1411.7923, 2014.PDFK. Zhang, Z. Zhang, Z. Li and Y. Qiao.Joint Face Detection and Alignment Using Multitask Cascaded Convolutional Networks, IEEE Signal Processing Letters, 2016.PDF