Package: challenge.uccs
This package implements the baseline algorithms and evaluation for the second version of the Unconstrained Face Detection and Open Set Recognition Challenge.
This package relies on the signal processing and machine learning librariesBobandCaffe.
For installation instructions and requirements ofBob, please refer to theBobweb page and theCaffeweb page.NoteDue to limitations ofBob, this package will run only under Linux and MacOS operating systems.
Particularly, it will not work under any version of Microsoft Windows, and maybe not under some other exotic operating systems.
If you experience problems with the installation, we would suggest to run the experiments in a virtual environment, e.g., usingOracle’s VirtualBox.
On request, we will generate a virtual image with this package pre-installed.DatasetThis package does not include the original image and protocol files for the competition.
Please register on theCompetition Websiteand download the UCCS dataset from there.Please extract all zip filesinto the same directory(the .zip files contain the appropriate directory structure).
This includes alltraining_*.zipandvalidation_*.zipfiles, as well as theprotocol.zipand possibly theSampleDataSet.zip.
The directory will be refereed to asYOUR-DATA-DIRECTORYbelow.InstallationThe installation of this package follows theBuildoutstructure.
First, you will need to installBobandCaffe, ideally into the same conda environment.
Otherwise, please point thePYTHONPATHvariable to whereCaffeis installed.
Before continuing, please make sure that you can successfully importbob.measureandcaffe:$ conda activate bob_py3
(bob_v3) $ python
>>> import bob.measure
>>> import caffeAfter installingBobandCaffeand extracting this package, please run the following command line to install this package and all of its requirements:buildoutThe installation procedure automatically generates executable files inside thebindirectory, which can be used to run the baseline algorithms or to evaluate the baselines (and your) algorithm.The networks that are used in our baselines can be downloaded from the Internet.
To run the face detector baseline, you will need to put themodeldirectory of theMTCNN v2face detector into the base directory of this package.
To run the face recognition baseline, you will need to go to theVGG v2website, download theVggface2_caffe_model.tar.gz, and extract thesenet50_ft.caffemodelandsenet50_ft.prototxtinto the samemodeldirectory.Running the BaselinesThere are two scripts to run the baseline, one for each part.Face DetectionThe first script is a face detection script, which will detect the faces in the validation (and test) set.
The baseline detector uses theMTCNN v2face detection module, where we had to lower the detection thresholds in order to detect most of the faces in the images – still around 20 percent of the (mostly difficult) faces could not be detected usingMTCNN v2.
Additionally, there are a lot of background areas that are mistakenly marked as faces.You can call the face detector baseline script using:./bin/baseline_detector.pyPlease refer to./bin/baseline_detector.py-hfor possible options.
Here is a subset of options that you might want/need to use/change:--data-directory: Specify the directory, where you have downloaded the UCCS dataset into.--result-file: The file to write the detection results into; this will be in the required format.--which-set: The set, for which the baseline detector should be run; possible values:training, validation, test, sample; default:validation.--verbose: Increase the verbosity of the script; using--verbose--verboseor-vvis recommended;-vvvwill write more information.--debug: Run only over the specified number of images; for debug purposes only.--display: Display the detected bounding boxes (green) and the ground truth (red); for debug purposes only.--parallel: Run in the given number of parallel processes; can speed up the processing tremendously.--gpus: Run the face detection usingCaffein GPU mode (this might not work depending on your setup ofCaffe); you can specify the device ids to use, typically only0; if not specified, runs in CPU modeOn a machine with 32 CPU cores, a good command line for the full baseline experiment would be:./bin/baseline_detector.py--data-directoryYOUR-DATA-DIRECTORY-vv--parallel32On a machine with 32 cores and two NVidia GPUs, a good command line for the full baseline experiment would be:./bin/baseline_detector.py--data-directoryYOUR-DATA-DIRECTORY-vv--parallel8--gpus0 1To run a small-scale experiment, i.e., to display the detected faces on 20 images, a good command line might be:./bin/baseline_detector.py--data-directoryYOUR-DATA-DIRECTORY-vvv--display--debug20NoteThe--displayoption requiresMatplotlibto be installed and set up properly.
Display does not work in parallel mode.By default, the face detection score file will be written to./results/UCCS-v2-detection-baseline-validation.txt.Face RecognitionFor face recognition, we utilize theMTCNN v2face detector to detect all the faces (see above) and make sure that the bounding boxes with the highest overlap to the ground truth labels are used.
For each of the detected bounding boxes in the training set, we enlarge the bounding box with a factor of 1.3 and extract features using thepool5/7x7_s1layer of theVGG v2recognition module, which we normalize to unit Euclidean length.
The template enrollment stage simply computes the average of the training set features for each of the known subjects.
To account for the fact that we have unknown subjects in the training and test set, we also enroll one gallery template including all known unknown subjects labeled as-1.For a probe image, we assume that we do not have any labels (this is what the test set will look like).
Hence, we run theMTCNN v2detector to detect faces (which will include several background regions).
For each bounding box, we extract theVGG v2feature as above, and computecosine similaritiesto all gallery templates including the unknown template.
The highest 10 similarities are stored into the score file – except when the unknown label (-1) is amongst the highest scores, in which case only the scores up to the-1label are stored.
This means that when the highest score is with the-1label, only one value is stored in the score file.You can call the face recognition baseline script using:./bin/baseline_recognizer.pyPlease refer to./bin/baseline_recognizer.py-hfor possible options.
Here is a subset of options that you might want/need to use/change:--data-directory: Specify the directory, where you have downloaded the UCCS dataset into--result-file: The file to write the recognition results into; this will be in the required format--verbose: Increase the verbosity of the script; using--verbose--verboseor-vvis recommended;-vvvwill write more information--temp-dir: Specify the directory, where temporary files are stored; these files will be computed only once and reloaded if present--force: Ignore existing temporary files and always recompute everything--debug: Run only over the specified number of identities; for debug purposes only; will modify file names of temporary files and result file--parallel: Run in the given number of parallel processes; can speed up the processing tremendouslyOn a machine with 32 CPU cores, a good command line would be:./bin/baseline_recognizer.py--data-directoryYOUR-DATA-DIRECTORY-vv--parallel32When usingCaffein GPU mode with two NVidia GPUs, a good command line would be:./bin/baseline_recognizer.py--data-directoryYOUR-DATA-DIRECTORY-vv--parallel2--gpus0 1By default, the face recognition score file will be written to./results/UCCS-v2-recognition-baseline-validation.txt.EvaluationThe provided evaluation scripts will be usable to evaluate the validation set only, not the test set (since the test set labels are not given to the participants).
You can use the evaluation scripts for two purposes:To plot the baseline results in comparison to your results.To make sure that your score file is in the desired format.If you are unable to run the baseline experiments on your machine, we provide the score files for the validation set on thecompetition website.Face Detection EvaluationAs the ground-truth is usually larger than the face, we do not punish bounding boxes that are smaller than the ground truth.
Therefore, the union (the denominator) takes into account only one fourth of the ground truth bounding box – or the intersection area, whichever is larger:\begin{equation*}
O(G,D) = \frac{|G \cap D|}{|G \cup D|} = \frac{G \cap D}{\max\{\frac{|G|}4, |G \cap D|\} + |D| - |G \cap D|}
\end{equation*}where\(|\cdot|\)is the area operator.
Hence, when the detected bounding box\(D\)covers at least a fourth of the ground-truth bounding box\(G\)and is entirely contained inside\(G\), an overlap of 1 is reached.The face detection is evaluated using the Free Receiver Operator Characteristic (FROC) curve, which plots the percentage of correctly detected faces over the total number of false alarms (detected background regions).
Different points on the FROC curve can be obtained for different detector confidence values.
This plot can be created using:./bin/evaluate_detector.pyThis script has several options, some of which need to be specified, see./bin/evaluate_detector.py-h:--data-directory: Specify the directory, where you have downloaded the UCCS dataset into--result-files: A list of all files containing detection (or recognition) results--labels: A list of labels for the algorithms; must be the same number and in the same order as--result-files--froc-file: The name of the output .pdf file containing the FROC plot; default isUCCS-v2-FROC.pdf--log-x: will plot the horizontal axis in logarithmic scale--only-present: will ignore any file for which no detection exists (for debug purposes only, i.e., when detector ran with the--debugoption)--verbose: Increase the verbosity of the script; using--verbose--verboseor-vvis recommendedTo plot the baseline FROC curve (which is shown on thecompetition website), execute:./bin/evaluate_detector.py--data-directoryYOUR-DATA-DIRECTORY--result-filesresults/UCCS-v2-detection-baseline-validation.txt--labelsBaseline--log-x-vvNoteIf you have run the face recognition baseline, you can also use the face recognition result file for plotting the FROC curve:./bin/evaluate_detector.py--data-directoryYOUR-DATA-DIRECTORY--result-filesresults/UCCS-v2-recognition-baseline-validation.txt--labelsBaseline--log-x-vvFace Recognition EvaluationOpen set face recognition is evaluated using the Detection and Identification Rate (DIR) curve, which plots the percentage of correctly detected and identified faces over the total number of false identifications.
Based on various values of the false identifications, several score thresholds are computed.
A face is said to be identified correctly if the recognition score is greater than the threshold and the correct identity has the highest recognition score for that face.
The number of correctly identified faces is computed, and divided by the total number of known faces.
For more details, please refer to[1].NoteBy default only rank 1 recognition is performed, but the evaluation can be done using any rank up to 10 (the upper bound of allowed labels per face).
Providing more than one identity label per face will increase the number of false alarms, and may only have an impact on higher rank evaluations.NoteUnknown identities or background regions labeled with label -1 or not labeled at all will be ignored (i.e., will not decrease performance).
Labeling an unknown identity or a background region with any other label than -1 will result in a false alarm – only the maximum score per bounding box will be considered.The DIR plot can be created using:./bin/evaluate_recognizer.pyAs usual, the script has several options, which are similar to./bin/evaluate_detector.pyabove, see./bin/evaluate_recognizer.py-hfor a complete list:--data-directory: Specify the directory, where you have downloaded the UCCS dataset into--result-files: A list of all files containing recognition results--labels: A list of labels for the algorithms; must be the same number and in the same order as--result-files--dir-file: The name of the output .pdf file containing the DIR plot; default isUCCS-v2-DIR.pdf--log-x: will plot the horizontal axis in logarithmic scale--only-present: will ignore any file for which no detection exists (for debug purposes only, i.e., when recognizer ran with the--debugoption)--verbose: Increase the verbosity of the script; using--verbose--verboseor-vvis recommended--rank: Use the given rank to plot the DIR curveTo plot the baseline Rank 1 DIR curve (which is shown on thecompetition website), execute:./bin/evaluate_recognizer.py--data-directoryYOUR-DATA-DIRECTORY--result-filesresults/UCCS-v2-recognition-baseline-validation.txt--labelsBaseline--log-x-vvTrouble ShootingIn case of trouble with running the baseline algorithm or the evaluation, please contact us via email under:opensetface@vast.uccs.edu[1]P. Jonathon Phillips, Patrick Grother, and Ross Micheals“Evaluation Methods in Face Recognition” inHandbook of Face Recognition, Second Edition, 2011.