Package: facerec-uccs
UCCS Watchlist ChallengeThis package utilizes the PyTorch framework to implement baseline algorithms and conduct evaluations for the third version of the Open-set Face Detection and Identification challenge.DatasetThis package does not include the original image and protocol files for the competition.
Please register on theCompetition Website. Upon confirming your registration, we will provide the data, including gallery and validation images along with their protocol files, in a zip file. Please extract zip filesinto a new directory nameddata(the .zip files contain the appropriate directory structure) within this package. After this, the directory structure should appear as follows:uccs-facerec-challenge/├──data/│├──gallery_images/││├──0001/│││├──0001_0.png│││├──...│││└──0001_9.png││├──...││└──1000│││├──validation_images/││├──jgsa451150sag15fou.jpg││├──...│││├──exclude_gallery_validation.pickle│├──gallery.csv│└──validation.csv│├──facerec/│├──configs/│├──script/│└──...│├──environment.yaml├──README├──setup.py└──...If you prefer to store this data in a different directory, you should modify or overwrite the--data_directoryoption in thefacerec/configs/baseline_config.yamlfile.  The test set images without any annotations will be distributed two weeks before the competition concludes, as stated on theCompetition Website.InstallationThe installation of this package follows via condaInstall via conda:gitclonehttps://github.com/AIML-IfI/uccs-facerec-challenge.gitcduccs-facerec-challengecondaenvcreate-fenvironment.yamlcondaactivateuccs-facerecpipinstall-e.ScriptsThere are four scripts in total, which can be found infacerec/script.
All scripts will be installed together with the installation of this package in order to run the baseline algorithms or to evaluate the baselines (and your) algorithm.
The networks employed in our baselines will be automatically downloaded when you run the corresponding script. Hence, there is no need for you to separately download these baseline algorithms which areMTCNNandMagFace.Each script is associated with its distinct set of parameters specified in the configuration.yamlfile, located atfacerec/configs/baseline_config.yaml.
In thebaseline_config.yamlfile, you can find all arguments, along with their default values, under their respective sections.
Nonetheless, there are four essential arguments placed at the top of the configuration file that are required for almost all scripts.Here are these options that you might want/need to use/change:--data_directory: Specify the directory, where you have downloaded the UCCS dataset into; default:data.--result_directory: Specify the directory, where the score files, evaluation curves, and embeddings will be stored; default:results. Note that this directory doesn't need to exist.--which_set: The set, for which the scripts should be run; possible values:validation, test; default:validation.--image_directory: Specify the directory, where the original images are stored; default:{data_directory}/{which_set}_images--gpu: The GPU index to run the detector/extractor on it, to run in CPU mode please specify-1; default:0.You can modify these parameters in the configuration file if desired, and it's also possible to overwrite any parameter on the command line. In this case, the path of the configuration file containing the arguments you wish to overwrite should be specified on the command line after calling the script. The detailed explanations for each script are provided within their corresponding sections below.Face DetectionThe first script is a face detection script, which will detect the faces in the validation (and test) set and write the results into a file.
The baseline detector uses the PyTorch implementation of theMTCNNface detection module, where we had to lower the detection thresholds in order to detect most of the faces in the images, but this leads to many background detections.You can easily call the face detector baseline script after successful installation using:baseline_detection.pyIf the baseline configuration file suits your environment, there's no need to specify the configuration file's path on the command line while calling the script. Simply running the script will automatically read the defaultbaseline_config.yamlfile.Please refer tobaseline_detection.py facerec/configs/baseline_config.yaml -hfor all possible options.Here are the options that you might want/need to use/change for this script:--detection.results: Specify the file to write the detection results into; default:{result_directory}/UCCS-detection-baseline-{which_set}.txt.--detection.parallel: Run in the given number (positive integer) of parallel processes; default:0Note thatTo achieve the same baseline detection results, the--detection.thresholds : [0.2,0.2,0.2]and--detection.max_detections: 50options should remain unchanged.Here is an example of how to overwrite any parameter in the configuration file using the command line:baseline_detection.pyfacerec/configs/baseline_config.yaml--data_directoryYOUR-DATA-DIRECTORY--detection.resultsYOUR-FILE-PATHFace RecognitionFor face recognition, we utilize theMagFace(the backbone of iResNet-100), a cutting-edge method in the field of face recognition to extract features from detected faces above.
The MagFace model and its weights don't require any manual downloading from the Internet; they are automatically downloaded when the script is called for the first time.You can easily call the face extractor baseline script after successful installation using:baseline_recognition.pyThe MagFace extracts features with the shape of (,512) for both probe and gallery.
For the validation, the script stores all information (detection scores, bounding boxes, and embeddings) of the faces based on the probe image.
For the gallery, it stores five facial landmarks and embeddings based on an identity. In conclusion, it compiles these dictionaries into .pth files for both sets, simplifying the process of generating the score file in the next step.Once the script execution is complete, the directory structure appears as follows:results/├──validation_embeddings/│├──probe-image1.pth││-Content(adict):{││'detection_scores':Arraywithshape(N,1),# where N is the number of detected faces in the image││'bboxes':Arraywithshape(N,4),││'embeddings':Arraywithshape(N,1)││}│││├──probe-image2.pth│├──...│├──gallery_embeddings/├──subject1.pth│-Content(adict):{│'landmarks':Arraywithshape(10,5,2),# each subject has 10 faces in the gallery│'embeddings':Arraywithshape(10,512)│}├──subject2.pth├──...If the baseline configuration file suits your environment, there's no need to specify the configuration file's path on the command line while running the script. Simply calling the script will automatically read thebaseline_config.yamlfile.
Please refer tobaseline_recognition.py facerec/configs/baseline_config.yaml -hfor all possible options.Here are the options that you might want/need to use/change for this script:--disable-gallery: When specified,  it will disable the gallery extraction with the given set at once; default:store_false--recognition.detection_file: The file containing the detected faces in the images; default:{result_directory}/UCCS-detection-baseline-{which_set}.txt.--recognition.result_dir: Specify the directory to store .pth files of the probe images/subjects; default:{result_directory}/{which_set}_embeddings--recognition.workers: How many sub-processes to use for data loading; default:4--recognition.batch_size_perImg: For validation (or test); it should be 1 because of the different number of faces in each image; for gallery, it can be multiples of 10; default:1. Even if you don't provide a proper value for the batch size, the script is able to handle it.Note thatTo achieve the same baseline recognition results, the--recognition.embedding_size: 512and--recognition.arch: iresnet100options should remain unchanged.Here is an example of how to overwrite any parameter in the configuration file using the command line:baseline_recognition.pyfacerec/configs/baseline_config.yaml--recognition.workersNUMBER-OF-WORKERS--recognition.result_dirYOUR-RESULT-DIRECTORYScoringThis script produces the desired score file for the last evaluation phase, applicable for both face detection and identification evaluations.
It initially reads gallery embeddings to establish enrollment, achieved by averaging 10 embeddings of the corresponding subject.
Therefore, each subject is represented by an array with a shape of (1,512) in the enrollment.
Following the enrollment, the script compares the embedding of each face in the probe images with those of 1000 subjects using cosine similarity.
Finally, it writes each similarity score of every subject along with their detection results (confidence scores and bounding boxes) into a file.Note thatIf your intention is only to participate in the face detection task, calling this script is unnecessary. Creating a file similar tobaseline_detection.pyis sufficient. Further details can be found on theCompetition Websiteregarding the expected format of score files.You can easily call the scoring script after successful installation using:scoring.pyIf the baseline configuration file suits your environment, there's no need to specify the configuration file's path on the command line while running the script. Simply calling the script will automatically read the defaultbaseline_config.yamlfile. Please refer toscoring.py facerec/configs/baseline_config.yaml -hfor all possible options.Here are the options that you might want/need to use/change for this script:--scoring.gallery: Specify the directory where the gallery embeddings (.pth files) are stored; default:{result_directory}/gallery_embeddings.--scoring.probe: Specify the directory where the probe embeddings (.pth files) are stored; default:{result_directory}/{which_set}_embeddings.--scoring.results: Specify the file to write the scores and detection results into; default:{result_directory}/UCCS-scoring-baseline-{which_set}.txt.Here is an example of how to overwrite any parameter in the configuration file using the command line:scoring.pyfacerec/configs/baseline_config.yaml--scoring.galleryGALLERY-EMBEDDINGS-DIRECTORY--scoring.probePROBE-EMBEDDINGS-DIRECTORYEvaluationThe provided evaluation script is usable to evaluate the validation set only, not the test set (since the test set labels will not be given to the participants).
By default, this script is capable of running the evaluation for both face detection and identification.
If you are unable to run the baseline experiments on your machine, we will provide the baseline score files for the validation set alongside the dataset when shared.You can use the evaluation script for two purposes:To plot the baseline results in comparison to your results.To make sure that your score file is in the desired format.You can easily call the evaluation script after successful installation using:evaluation.pyIf the baseline configuration file suits your environment, there's no need to specify the configuration file's path on the command line while running the script. Simply calling the script will automatically read the defaultbaseline_config.yamlfile.
Please refer toevaluation.py facerec/configs/baseline_config.yaml -hfor all possible options.Here are the main options that you might want/need to use/change for this script:--tasks: Specify the tasks that will perform in this evaluation; possible values:detection, recognition; default :['detection', 'recognition'].--eval.exclude_gallery:  Specify the file where gallery face IDs are stored to exclude them from the results; default ={data_directory}/exclude_gallery_{which_set}.pickle. Note that since the gallery faces are cropped from the dataset, they will be excluded from the results. This list of gallery face IDs will be shared along with the dataset.--eval.linear: If specified, plots will be in linear, otherwise semilogx; default :False.Note thatIf you plan to participate in both challenges, the face recognition score file can be used for evaluating both the detection and the recognition experiment. Therefore, it is enough to only submit the desired identification score file. More details can be found in theCompetition Website.DetectionIn the face detection evaluation, the script compares detected bounding boxes, utilizing the standard IOU metric with a threshold of0.5, against the ground truth.
Only the detection box with the highest overlap can be considered a true positive, while others are penalized.
The evaluation employs the Free Receiver Operator Characteristic (FROC) curve, ploting the Detection Rate (percentage of correctly detected faces) against False Detection per Image (detected background regions).
False Detection per Image is calculated by dividing the misdetections by the number of probe images.
Different points on the FROC curve can be obtained for different detector confidence values.You can easily call the evaluation script for only detection task after successful installation using:evaluation.pyfacerec/configs/baseline_config.yaml--tasksdetectionHere are the options that you might want/need to use/change for this detection task:--eval.detection.files: A list of score file(s) containing detection results; default :['{result_directory}/UCCS-detection-baseline-{which_set}.txt']. For comparison, different detection score files can be added.--eval.detection.labels: A list of label(s) for the algorithms; must be the same number and in the same order as--eval.detection.files; default:['mtcnn'].--eval.detection.froc: The .pdf file where FROC curve will be written into; default :{result_directory}/UCCS-FROC-{which_set}.pdf.Note thatTo achieve the same baseline FROC curve on the validation, the--eval.iou: 0.5option should remain unchanged.Here is an example of how to overwrite any parameter in the configuration file using the command line:evaluation.pyfacerec/configs/baseline_config.yaml--tasksdetection--eval.detection.filesFILE1FILE2--eval.detection.labelsLABEL1LABEL2RecognitionDuring the evaluation of face recognition models, faces will be either assigned to an identity or rejected based on these similarity scores.
All unique similarity score values in the score file will be threshold values for matching.
A face is counted correctly identified if the recognition score surpasses the threshold, and the correct identity possesses the highest recognition score for that particular face.
Providing high scores for unknown identities or misdetections, which indicate a false match with a gallery identity, will result in penalties.The evaluation utilizes a modified version of the Detection and Identification Rate (DIR) curve on rank 1, also known as the Open-Set ROC curve that computes True Positive Identification Rates (TPIR) over False Positive Identification Rates (FPIR).
Since the FPIR is dependent on the number of detected faces, we make slight modifications to the false alarms axis of this curve by dividing it by the number of probe images, leading to the False Positive Identification per Image.
This x-axis is in a logarithmic scale, representing non-rejected unknown faces and misdetections.
To prevent an increase in False Identifications, these unknown faces or misdetections should have a similarity score lower than the threshold specified for the points on the curve.
For more details, please refer to [1].You can easily call the evaluation script for only the recognition task after successful installation using:evaluation.pyfacerec/configs/baseline_config.yaml--tasksrecognitionHere are the options that you might want/need to use/change for this recognition task:--eval.recognition.files: A list of score file(s) containing recognition results; default :['{result_directory}/UCCS-scoring-baseline-{which_set}.txt']. For comparison, different recognition score files can be added.--eval.recognition.labels: A list of the label(s) for the algorithms; must be the same number and in the same order as--eval.recognition.files; default:['MagFace'].--eval.recognition.rank: Plot DIR curves for the given rank; default :1.--eval.recognition.oroc: The .pdf file where O-ROC curve will be written into; default :{result_directory}/UCCS-OROC-{which_set}.pdf.Note thatTo achieve the same baseline O-ROC curve on the validation, the--eval.iou: 0.5and--eval.rank.recognition: 1options should remain unchanged.Here is an example of how to overwrite any parameter in the configuration file using the command line:evaluation.pyfacerec/configs/baseline_config.yaml--tasksrecognition--eval.recognition.filesFILE1FILE2--eval.recognition.labelsLABEL1LABEL2Trouble ShootingIn case of trouble with running the baseline algorithm or the evaluation, please contact us via email:furkan.kasim@uzh.ch[1]P. Jonathon Phillips, Patrick Grother, and Ross Micheals"Evaluation Methods in Face Recognition" inHandbook of Face Recognition, Second Edition, 2011.