Package: facetorch
facetorchHugging Face Space demo app 🤗Google Colab notebook demoUser Guide,Documentation,ChatGPT facetorch guideDocker Hub(GPU)Facetorch is a Python library that can detect faces and analyze facial features using deep neural networks. The goal is to gather open sourced face analysis models from the community, optimize them for performance using TorchScript and combine them to create a face analysis tool that one can:configure usingHydra(OmegaConf)reproduce withconda-lockandDockeraccelerate on CPU and GPU withTorchScriptextend by uploading a model file to Google Drive and adding a config yaml file to the repositoryPlease, use the library responsibly with caution and follow theethics guidelines for Trustworthy AI from European Commission.
The models are not perfect and may be biased.InstallPyPIpipinstallfacetorchCondacondainstall-cconda-forgefacetorchUsagePrerequisitesDockerDocker ComposeDocker Compose provides an easy way of building a working facetorch environment with a single command.Run docker exampleCPU:docker compose run facetorch python ./scripts/example.pyGPU:docker compose run facetorch-gpu python ./scripts/example.py analyzer.device=cudaCheckdata/outputfor resulting images with bounding boxes and facial 3D landmarks.(Apple Mac M1) Use Rosetta 2 emulator in Docker Desktop to run the CPU version.ConfigureThe project is configured by files located inconfwith the main file:conf/config.yaml. One can easily add or remove modules from the configuration.ComponentsFaceAnalyzer is the main class of facetorch as it is the orchestrator responsible for initializing and running the following components:Reader - reads the image and returns an ImageData object containing the image tensor.Detector - wrapper around a neural network that detects faces.Unifier - processor that unifies sizes of all faces and normalizes them
between 0 and 1.Predictor dict - set of wrappers around neural networks trained to analyze facial features.Utilizer dict - set of wrappers around any functionality that requires the output of neural networks e.g. drawing bounding boxes or facial landmarks.Structureanalyzer
    ├── reader
    ├── detector
    ├── unifier
    └── predictor
            ├── embed
            ├── verify
            ├── fer
            ├── au
            ├── va
            ├── deepfake
            └── align
    └── utilizer
            ├── align
            ├── draw
            └── saveModelsDetector|     model     |   source  |   params  |   license   | version |
| ------------- | --------- | --------- | ----------- | ------- |
|   RetinaFace  |  biubug6  |   27.3M   | MIT license |    1    |biubug6code:Pytorch_Retinafacepaper:Deng et al. - RetinaFace: Single-Shot Multi-Level Face Localisation in the WildPredictorFacial Representation Learning (embed)|       model       |   source   |  params |   license   | version |  
| ----------------- | ---------- | ------- | ----------- | ------- |
|  ResNet-50 VGG 1M |  1adrianb  |  28.4M  | MIT license |    1    |1adrianbcode:unsupervised-face-representationpaper:Bulat et al. - Pre-training strategies and datasets for facial representation learningNote:include_tensorsneeds to be True in order to include the model prediction in Prediction.logitsFace Verification (verify)|       model      |   source    |  params  |      license       | version |  
| ---------------- | ----------- | -------- | ------------------ | ------- |
|    MagFace+UNPG  | Jung-Jun-Uk |   65.2M  | Apache License 2.0 |    1    |
|  AdaFaceR100W12M |  mk-minchul |    -     |     MIT License    |    2    |Jung-Jun-Ukcode:UNPGpaper:Jung et al. - Unified Negative Pair Generation toward Well-discriminative Feature Space for Face Recognition(FAR=0.01)Note:include_tensorsneeds to be True in order to include the model prediction in Prediction.logitsmk-minchulcode:AdaFacepaper:Kim et al. - AdaFace: Quality Adaptive Margin for Face Recognition<<< badges represent models trained on smaller WebFace 4M datasetNote:include_tensorsneeds to be True in order to include the model prediction in Prediction.logitsFacial Expression Recognition (fer)|       model       |      source    |  params  |       license      | version |  
| ----------------- | -------------- | -------- | ------------------ | ------- |
| EfficientNet B0 7 | HSE-asavchenko |    4M    | Apache License 2.0 |    1    |
| EfficientNet B2 8 | HSE-asavchenko |   7.7M   | Apache License 2.0 |    2    |HSE-asavchenkocode:face-emotion-recognitionpaper:Savchenko - Facial expression and attributes recognition based on multi-task learning of lightweight neural networksB2B0B0Facial Action Unit Detection (au)|        model        |   source  |  params |       license      | version |  
| ------------------- | --------- | ------- | ------------------ | ------- |
| OpenGraph Swin Base |  CVI-SZU  |   94M   |     MIT License    |    1    |CVI-SZUcode:ME-GraphAUpaper:Luo et al. - Learning Multi-dimensional Edge Feature-based AU Relation Graph for Facial Action Unit Recognition! Does not work with CUDA > 12.0Facial Valence Arousal (va)|       model       |   source   |  params |   license   | version |
| ----------------- | ---------- | ------- | ----------- | ------- |
|  ELIM AL AlexNet  | kdhht2334  |  2.3M   | MIT license |    1    |kdhht2334code:ELIMpaper:Kim et al. - Optimal Transport-based Identity Matching
for Identity-invariant Facial Expression RecognitionDeepfake Detection (deepfake)|         model        |      source      |  params  |   license   | version |
| -------------------- | ---------------- | -------- | ----------- | ------- |
|    EfficientNet B7   |     selimsef     |   66.4M  | MIT license |    1    |selimsefcode:dfdc_deepfake_challengechallenge:Seferbekov - Deepfake Detection Challenge 1st place solutionFace Alignment (align)|       model       |      source      |  params  |   license   | version |
| ----------------- | ---------------- | -------- | ----------- | ------- |
|    MobileNet v2   |     choyingw     |   4.1M   | MIT license |    1    |choyingwcode:SynergyNetchallenge:Wu et al. - Synergy between 3DMM and 3D Landmarks for Accurate 3D Facial GeometryNote:include_tensorsneeds to be True in order to include the model prediction in Prediction.logitsModel downloadModels are downloaded during runtime automatically to themodelsdirectory.
You can also download the models manually from apublic Google Drive folder.Execution timeImage test.jpg (4 faces) is analyzed (including drawing boxes and landmarks, but not saving) in about 486ms and test3.jpg (25 faces) in about 1845ms (batch_size=8) on NVIDIA Tesla T4 GPU once the default configuration (conf/config.yaml) of models is initialized and pre heated to the initial image size 1080x1080 by the first run. One can monitor the execution times in logs using the DEBUG level.Detailed test.jpg execution times:analyzer
    ├── reader: 27 ms
    ├── detector: 193 ms
    ├── unifier: 1 ms
    └── predictor
            ├── embed: 8 ms
            ├── verify: 58 ms
            ├── fer: 28 ms
            ├── au: 57 ms
            ├── va: 1 ms
            ├── deepfake: 117 ms
            └── align: 5 ms
    └── utilizer
            ├── align: 8 ms
            ├── draw_boxes: 22 ms
            ├── draw_landmarks: 7 ms
            └── save: 298 msDevelopmentRun the Docker container:CPU:docker compose -f docker-compose.dev.yml run facetorch-devGPU:docker compose -f docker-compose.dev.yml run facetorch-dev-gpuAdd predictorPrerequisitesfile of the TorchScript modelID of the Google Drive model filefacetorchforkFacetorch works with models that were exported from PyTorch to TorchScript. You can applytorch.jit.tracefunction to compile a PyTorch model as a TorchScript module. Please verify that the output of the traced model equals the output of the original model.The first models are hosted on mypublic Google Drive folder. You can either send the new model for upload to me, host the model on your
Google Drive or host it somewhere else and add your own downloader object to the codebase.ConfigurationCreate yaml fileCreate new folder with a short name of the task in predictor configuration directory/conf/analyzer/predictor/following the FER example in/conf/analyzer/predictor/fer/Copy the yaml file/conf/analyzer/predictor/fer/efficientnet_b2_8.yamlto the new folder/conf/analyzer/predictor/<predictor_name>/Change the yaml file name to the model you want to use:/conf/analyzer/predictor/<predictor_name>/<model_name>.yamlEdit yaml fileChange the Google Drive file ID to the ID of the model.Select the preprocessor (or implement a new one based on BasePredPreProcessor) and specify it's parameters e.g. image size and normalization in the yaml file
to match the requirements of the new model.Select the postprocessor (or implement a new one based on BasePredPostProcessor) and specify it's parameters e.g. labels in the yaml file to match
the requirements of the new model.(Optional) Add BaseUtilizer derivative that uses output of your model to perform some additional actions.Configure testsAdd a new predictor to the mainconfig.yamland alltests.config.n.yamlfiles. Alternatively, create a new config file e.g.tests.config.n.yamland add it to the/tests/conftest.pyfile.Write a test for the new predictor in/tests/test_<predictor_name>.pyTest and submitRun linting:black facetorchAdd the new predictor to the README model table.Update CHANGELOG and versionSubmit a pull request to the repositoryUpdate environmentCPU:Add packages with corresponding versions toenvironment.ymlfileLock the environment:conda lock -p linux-64 -f environment.yml --lockfile conda-lock.yml(Alternative Docker) Lock the environment:docker compose -f docker-compose.dev.yml run facetorch-lockInstall the locked environment:conda-lock install --name env conda-lock.ymlGPU:Add packages with corresponding versions togpu.environment.ymlfileLock the environment:conda lock -p linux-64 -f gpu.environment.yml --lockfile gpu.conda-lock.yml(Alternative Docker) Lock the environment:docker compose -f docker-compose.dev.yml run facetorch-lock-gpuInstall the locked environment:conda-lock install --name env gpu.conda-lock.ymlRun tests + coverageRun tests and generate coverage:pytest tests --verbose --cov-report html:coverage --cov facetorchGenerate documentationGenerate documentation from docstrings using pdoc3:pdoc --html facetorch --output-dir docs --force --template-dir pdoc/templates/ProfilingRun profiling of the example script:python -m cProfile -o profiling/example.prof scripts/example.pyOpen profiling file in the browser:snakeviz profiling/example.profAcknowledgementsI want to thank the open source code community and the researchers who have published the models. This project would not be possible without their work.Logo was generated usingDeepAI Text To Image API