Package: xfacereclib.paper.IET2015
This package provides the source code to run the experiments published in the paperImpact of Eye Detection Error on Face Recognition Performance.
It relies on theFaceRecLibto execute the face recognition experiments, which in turn uses the face recognition algorithms and the database interface ofBob.NoteCurrently, this package only works in Unix-like environments and under MacOS.
Due to limitations of theBoblibrary, MS Windows operating systems are not supported.
We are working on a port ofBobfor MS Windows, but it might take a while.
In the meanwhile you could use ourVirtualBoximages that can be downloadedhere.When you use this source code in a scientific publication, we would be happy if you would cite:@article{Dutta2015,
  author = "Abhishek Dutta and Manuel G\"unther and Laurent El Shafey and S\'ebastien Marcel",
  title = "Impact of Eye Detection Error on Face Recognition Performance",
  year = 2015
  journal = {IET Biometrics},
  issn = {2047-4938},
  url = {http://digital-library.theiet.org/content/journals/10.1049/iet-bmt.2014.0037},
  pdf = {http://publications.idiap.ch/downloads/papers/2015/Dutta_IETBIOMETRICS_2014.pdf}
}InstallationThis package uses severalBoblibraries, which will be automatically installed locally using the command lines as listed below.
However, in order for theBobpackages to compile, certainDependenciesneed to be installed.This packageThe installation of this package relies on theBuildOutsystem.
By default, the command line sequence:$ ./python bootstrap-buildout.py
$ ./bin/buildoutshould download and install all required packages ofBobin the versions that we used to produce the results.
Other versions of the packages might generate sightly different results.
To use the latest versions of allBobpackages, please remove the strict version numbers that are given in thebuildout.cfgfile in the main directory of this package.Image DatabaseThe experiments are run on an external image database.
We do not provide the images from the database themselves.
Hence, please contact the database owners to obtain a copy of the images.
The Multi-PIE database used in our experiments can be downloadedhere.NoteUnfortunately, the Multi-PIE database is not free of charge.
If you do not have a copy of the database yet, and you are not willing to pay for it, you cannot reproduce the results of the paper directly.
Nevertheless, you can use other databases, some of which are free of charge.
A complete list of supported databases and their according evaluation protocols can be found in theFaceRecLibdocumentation.Important!After downloading the databases, you will need to tell our software, where it can find them by changing theconfiguration file.
In particular, please update theMULTIPIE_IMAGE_DIRECTORYinxfacereclib/paper/IET2015/configuration/database.py.Unpacking the AnnotationsAfter the database is set up correctly, you’ll need to unpack the eye annotations that are used in the experiments.
Please run the script:$ ./bin/unpack_annotationsto extract the annotations in the desired directory structure.
If you want, you can specify another directory to unpack the annotations (see./bin/unpack_annotations.py--help), but all other functions and configurations will have their defaults set according to the default directory.Testing your InstallationAfter you have set up the database, you should be able to run our test suite:$ ./bin/nosetestsPlease make sure that all tests pass.TODO:Implement tests.Getting helpIn case anything goes wrong, please feel free to open a new ticket in ourGitHubpage, start a new discussion in ourMailing Listor send an email tomanuel.guenther@idiap.ch.Recreating the Results of thePaperAfter successfully setting up the database, you are now able to run the face recognition experiments as explained in thePaper.
Particularly, you will be able to reproduce Figure 4, Figure 7 and Figure 13.
Be aware that we were running more than 1000 individual face recognition experiments, each of which used a slightly different experiment configuration.The Experiment ConfigurationThe face recognition experiment are run using theFaceRecLib.
In total, we are testing five different face recognition algorithms, each of which uses thedefault configurationfrom theFaceRecLib:eigenfaces: a PCA is trained on pixel gray values, and the projected features are compared with Euclidean distance.fisherfaces: a combined PCA + LDA matrix is trained on pixel gray values, and the projected features are compared with Euclidean distance.gabor-jet: Gabor jets are extracted at grid locations in the image and compared with a Gabor-phase-based similarity function.lgbphs: extended local Gabor binary pattern histogram sequences are extracted from image blocks, and the histograms are compared with histogram intersection.isv: DCT features are extracted from image blocks and modeled with a Gaussian mixture model and an additional inter-session variability model, and the score is computed as a likelihood ratio.As input, all these algorithms expect images, where the face is extracted and aligned, so that the eye centers are always placed on the same location in the image.
For this alignment procedure, labeled eye locations must be available.
The main focus of this paper isnoton the face recognition algorithms themselves, but on how they perform in case that the eye locations are slightly misplaced, as it might happen in both manual and automatic annotations.Running the ExperimentsFor convenience, we have generated a wrapper script that allows to run a set of face recognition experiments in sequence – or even in parallel, see below.
This wrapper scriptabusesone functionality of theFaceRecLib, namely theparameter testing, which is an easy way to perform a grid search on a set of parameters.
For our purposes, these parameters are:Figure 4: the eye position shifts in horizontal and vertical direction, as well as the rotation angle.Figure 7: the standard deviations of the Normal distributed shifts of eye positions in horizontal and vertical direction, as well as a random seed.The according configurations are given infixed_perturbation.py(Figure 4) andrandom_perturbation.py(Figure 7).
There, you can find the setup as it was used to generate the according plots, but in case you want to run only a sub-set of experiments, you can reduce the parameters in each list.The experiments can be run using the./bin/parameter_test.pyscript.
This script has several options, the most important of which are:--configuration-file: the configuration file that contains the parameters that we want to test.
For our experiments, these are the two filesfixed_perturbation.py(Figure 4) andrandom_perturbation.py(Figure 7).--database: the database that should be used in the experiments, which will bemultipie-min all cases.--executable: the (pythonic) name of the face verification function that will be executed.
Since we had to modify the default script a bit, our script needs to be specified (see below).--sub-directory: the name of a directory (will be created on need), where all experiments for the given configuration file are stored.--grid: a name of a grid configuration to run algorithms in parallel (see below).--verbose: Print out additional information or debug information during the execution of the experiments.
The--verboseoption can be used several times, increasing the level to Warning (1), Info (2) and Debug (3). By default, only Error (0) messages are printed.
The Info (aka-vv) option is recommended.--dry-run: Use this option to print the calls to theFaceRecLibwithout executing them.
Again, it is recommended to use this flag once, i.e., to check that everything is correct before running the experiments.Additionally, parameters can be passed directly to the./bin/faceverify.pyscript from theFaceRecLib.
Please use a--to separate parameters for./bin/faceverify.pyform parameters for./bin/parameter_test.py.
Useful parameters might be the--result-directoryand the--temp-directoryoptions.
For a complete list of options, please check./bin/faceverify.py--help.Finally, the command lines to run the experiments for Figures 4 and 7, call:$ ./bin/parameter_test.py --configuration-file fixed_perturbation.py --database multipie-m --sub-directory fixed --executable xfacereclib.paper.IET2015.script.faceverify -- --temp-directory [YOUR_TEMP_DIRECTORY] --result-directory [YOUR_RESULT_DIRECTORY]

$ ./bin/parameter_test.py --configuration-file random_perturbation.py --database multipie-m --sub-directory random --executable xfacereclib.paper.IET2015.script.faceverify -- --temp-directory [YOUR_TEMP_DIRECTORY] --result-directory [YOUR_RESULT_DIRECTORY]The last set of experiments, i.e., to regenerate Figure 13 can be run using the./bin/annotation_typesscript.
Again, this script has a set of options, most of which have proper default values:--image-directory: the base directory of the Multi-PIE database; needs to be specified.--annotation-directory: the base directory, where the annotations have been extracted to.--algorithms: a list of algorithms that should be tested; by default all five algorithms are run.--world-types: a list of annotation types, which should serve to train the algorithms and to enroll the models with.--probe-types: a list of annotation types, which should be probed against the enrolled models.Again, the same--verboseoption and options passed to the./bin/faceverify.pyscript exists.
Hence, the last set of experiments to be run can be started with:$ ./bin/annotation_types --image-directory [MULTIPIE_IMAGE_DIRECTORY] -vv -- --temp-directory [YOUR_TEMP_DIRECTORY] --result-directory [YOUR_RESULT_DIRECTORY]Parallel ExecutionSince the two command lines above execute more than 1000 individual face recognition algorithms, you might want to run them in parallel.
For this purpose, you can use the--gridoption of the./bin/parameter_test.pyscript.
This will trigger the usage ofGridTK, a tool originally developed to submit and monitor jobs in an SGE processing farm.
If you have access to such a farm, you can use the--gridsgeoption to submit the experiments to the SGE grid (you might need to set up the SGE configuration in the grid configuration filexfacereclib/paper/IET2015/configuration/grid.py, in thefacereclib/utils/grid.pyof theFaceRecLibor in theGridTKitself).On the other hand, when you have a powerful machine with lots of processing units, you can use the--gridlocaloption.
This will submit jobs to the “local” queue, which you have to start them manually by:$ ./bin/jman --local --database [DIR]/submitted.sql3 -vv run-scheduler --parallel [NUMBER_OF_SLOTS] --die-when-finishedPlease refer to theGridTKmanual for more details.NoteWhen submitting to the either the local queue or the SGE, several job databases calledsubmitted.sql3are stored in sub-directories of thegrid_dbdirectory.
You can use./bin/jman--database[DIR]/submitted.sql3listto see the current status of the jobs stored in the given database.
Of course, you can also use the default SGE tools (such asqstat) to check the statuses of the jobs.WarningFor the random experiment, please do not use more than one parallel job to preprocess the images.
Otherwise, the random seed might be applied several times, leading to inexact results.NoteThe same--gridoption can be used for the./bin/annotation_typesscript.
Here, only onesubmitted.sql3file is written, in the current directory.Evaluating the ExperimentsAfter all experiments have finished successfully, the resulting score files can be evaluated.
The figures in the paper were generated using a mix of python and R scripts, i.e., to make them look more beautiful.
However, for this package we will plot the figures solely using matplotlib.
The./bin/plot_resultsscript can be used to create the plots similar to the ones in Figures 4, 7 and 13.
Additionally, it will write.csvfiles containing the exact numbers, i.e., the Figures in in thePaperrely on these files.As usual, the./bin/plot_resultshas a list of command line options, most of which have proper default values:--scores-directory: the base directory, where the score files have been produced.--experiments: a list of experiments to evaluate.
By default, all three experiments are evaluated.--algorithms: a list of algorithms to evaluate.
By default, all five algorithms are evaluated.Some more options are available, see./bin/plot_results--helpfor a complete list.
Hence, to produce all three plots from Figures 4, 7, and 13, simply call:$ ./bin/plot_results -vv --scores-directory [YOUR_RESULT_DIRECTORY]Afterward, the plots can be found in theplotsdirectory.
For Figure 4, they are calledHTER_fixed.pdfandAUC_fixed.pdf, while for Figure 7 they areHTER_random.pdfandAUC_random.pdf.
The HTER plots should be identical to the ones found in thePaper.
The AUC plots have a different color coding than in thePaper, but the contents are identical.
Finally, the fileplots/ROCs.pdfcontains the ROC curves of Figure 13, except that the FAR range is slightly higher.