Package: fer-pytorch
FER-pytorchFacial expression recognition package built on Pytorch andFER+ dataset from Microsoft.Installationpip install fer-pytorchTrainingTraining is done using the synergy ofPytorch LightningandHydrapackages for setting training loops and configs correspondingly.
In order to run training you should clone the repo andInstall dependenciespip install -r requirements/requirements-dev.txtDefine environmental variablesJust runexport PYTHONPATH="$PWD"from the root directory and it would be enough
to run the code from the root. You can check the PYTHONPATH withecho $PYTHONPATHcommand.Download databash get_data.shThis bash command will download the archive with the data and unpack it into the necessary directories in such a way
that everything is ready for training.Define configThe config files are located infer-pytorch/confdirectory. To see all the parameters
and their default values runpython fer_pytorch/run_trainer.py -hRun trainingTraining with default parameter values:python fer_pytorch/run_trainer.pyThanks to Hydra all the parameters set in config files can be changed
directly from the command line while running the script:Example with change of model from resnet34 to resnet18:python fer_pytorch/run_trainer.py model.model_name="resnet18"Example with change of number of epochs:python fer_pytorch/run_trainer.py trainer.trainer_params.max_epochs=100By default the output is saved tooutput/directory. If you wish to set
the path to output, runpython fer_pytorch/run_trainer.py hydra.run.dir=path_to_outputInferenceImport inference classimport cv2
from fer_pytorch.fer import FER

fer = FER()Initialize the modelThere are 2 options:fer.get_pretrained_model(model_name): download the ready-to-use pretrained on FER+ dataset weights from the github
page of the package and initialize the model automatically. The list of available names are given infer_pytorch/pre_trained_models.pyfile as the keys ofmodelsdictionary. For example,fer.get_pretrained_model("resnet34").fer.load_user_weights(model_name, path_to_weights):  with this option you can load your own weights
that are stored locally.Inference on an imageBasic predictionimg = cv2.imread("tests/test_images/happy.jpg")
result = fer.predict_image(img)Sample output:[{'box': [295.90848, 87.36073, 463.75354, 296.00055],
'emotions': {'neutral': 0.00033704843, 'happiness': 0.98931086, 'surprise': 0.00018355528, 'sadness': 0.0026534477, 'anger': 0.0054451805, 'disgust': 0.0019571118, 'fear': 0.000112833266}}]Get only top emotion:result = fer.predict_image(frame, show_top=True)Sample output:[{'box': [295.90848, 87.36073, 463.75354, 296.00055], 'top_emotion': {'happiness': 0.98931086}}]In order to save the output image, just set the output path:result = fer.predict_image(frame, show_top=True, path_to_output="result.jpg")NoteIf there is more than one person on an image, this method will provide the results for all,
whose faces are detected by the MTCNN detector. However, this is not the case for the methods
described below (for list of images and video files). Instead, the inference methods for the list
of images ad video files will return results for people with the largest bounding box around
their faces. It is made for convenience and performance purposes since the mentioned methods
output the json file with the results, so it would be necessary to track the people to
produce the correct json files for corresponding detected people.Inference on a list of imagesInference on a folder with images:result_json_list = fer.predict_list_images(
    path_to_input="tests/test_images",
    path_to_output="tests/output_images",
    save_images=True
)Outputs the json with results and optionally the processed images in a separate directory.It is also possible to read the result json with pandas leveraging FER class method:result_df = FER.json_to_pandas("tests/output_images/result.json")
print(result_df.head())Inference on a video filefer.analyze_video(
    path_to_video="tests/test_videos/test_video.mp4",
    path_to_output="tests/test_video",
    save_video=True
)Outputs the json with results and optionally the processed video in a separate directory.Just like in the previous case it is also possible to read the result json with
pandas for further analysis leveraging FER class method:df = FER.json_to_pandas("tests/test_video/result.json")
print(df.head())Inference on the test part of the FER datasetGet the dictionary with accuracy and f1 score for the test part of the FER+ dataset:result_dict = fer.test_fer(
    path_to_dataset = "fer_pytorch/dataset",
    path_to_csv = "fer_pytorch/dataset/new_test.csv",
    batch_size = 32,
    num_workers = 8,
)
print(result_dict)Output:{'accuracy': 0.83, 'f1': 0.83}Inference with the web cameraTo run the model on the stream from the web camera and show the results in real-time just runfer.run_webcam()Web applicationIn order to demonstrate the fer-pytorch package the web application has been developed
usingStreamlitand deployed toHerokuvia Docker image.Link to the github page of the app:https://github.com/Emilien-mipt/fer-webappLink to the app:https://ferpytorch-webapp.herokuapp.com/