Package: xfacereclib.paper.IET2014
This package provides the source code to run the experiments published in the paperScore Calibration in Face Recognition.
It relies on theFaceRecLibto execute the face recognition experiments, and onBobto compute the calibration experiments.NoteCurrently, this package only works in Unix-like environments and under MacOS.
Due to limitations of theBoblibrary, MS Windows operating systems are not supported.
We are working on a port ofBobfor MS Windows, but it might take a while.
In the meanwhile you could use ourVirtualBoximages that can be downloadedhere.InstallationThe installation of this package relies on theBuildOutsystem. By default, the command line sequence:$ ./python bootstrap.py
$ ./bin/buildoutshould download and install all requirements, including theFaceRecLib, the database interfacesxbob.db.scface,xbob.db.mobioand all their required packages.
There are a few exceptions, which are not automatically downloaded:BobThe face recognition experiments rely on the open source signal-processing and machine learning toolboxBob.
To installBob, please visithttp://www.idiap.ch/software/boband follow the installation instructions.
Please verify that you have at least version 1.2.0 of Bob installed.
If you have installed Bob in a non-standard directory, please open the buildout.cfg file from the base directory and set theprefixesdirectory accordingly.NoteThe experiments that we report in thePaperwere generated with Bob version 1.2.1 andFaceRecLibversion 1.2.1.
If you use different versions of either of these packages, the results might differ slightly.
For example, we are aware that, due to some initialization differences, the results using Bob 1.2.0 and 1.2.1 are not identical, but similar.Image DatabasesThe experiments are run on external image databases.
We do not provide the images from the databases themselves.
Hence, please contact the database owners to obtain a copy of the images.
The two databases used in our experiments can be downloaded here:SCface [scface]:http://www.scface.orgMOBIO [mobio]:http://www.idiap.ch/dataset/mobioNoteFor the MOBIO database, you need to sign the EULA to get access to the data – the process is explained under the above MOBIO link.
After signing the EULA, please download the image filesIMAGES_PNG.tar.gzand the annotation filesIMAGE_ANNOTATIONS.tar.gzand extract the archives into directories of your choice.Important!After downloading the databases, you will need to tell our software, where it can find them by changing theconfiguration files.
In particular, please update thescface_directoryinxfacereclib/paper/IET2014/database_scface.py, as well asmobio_image_directoryandmobio_annotation_directoryinxfacereclib/paper/IET2014/database_mobio.py.
Please let all other configuration parameters unchanged as this might influence the face recognition experiments and, hence, the reproducibility of the results.Getting helpIn case anything goes wrong, please feel free to open a new ticket in ourGitLabpage, or send an email tomanuel.guenther@idiap.ch.Recreating the results of thePaperAfter successfully setting up the databases, you are now able to run the face recognition and calibration experiments as explained in thePaper.The experiment configurationThe face recognition experiment are run using theFaceRecLib, but for convenience there exists a wrapper script that set up the right parametrization for the call to theFaceRecLib.
The configuration files that are used by theFaceRecLib, which contain all the parameters of the experiments, can be found in thexfacereclib/paper/IET2014/directory.
Particularly, thexfacereclib/paper/IET2014/dct_mobio.pyandxfacereclib/paper/IET2014/isv_mobio.pyfiles contain the configuration for the DCT block features and the ISV algorithm as described in thePaper.
Accordingly,Running the experimentsThis script can be found inbin/iet2014_face_recog.py.
It requires some command line options, which you can list using./bin/iet2014_face_recog.py--help.
Usually, the command line options have a long version (starting with--) and a shortcut (starting with a single-), here we use only the long versions:--temp-directory: Specify a directory where temporary files will be stored (default:temp). This directory can be deleted after all experiments ran successfully.--result-directory: Specify a directory where final result files will be stored (default:results). This directory is required to evaluate the experiments.--databases: Specify a list of databases that you want your experiments to run on. Possible values arescfaceandmobio. By default, experiments on both databases are executed.--protocols: Specify a list of protocols that you want to run. Possible values arecombined,close,mediumandfarfor databasescface, andmaleandfemaleformobio. By default, all protocols are used.--combined-zt-norm: Execute the face recognition experiments on the SCface database with combined ZT-norm cohort.--verbose: Print out additional information or debug information during the execution of the experiments. The--verboseoption can be used several times, increasing the level to Warning (1), Info (2) and Debug (3). By default, only Error (0) messages are printed.--dry-run: Use this option to print the calls to theFaceRecLibwithout executing them.Additionally, you can pass options directly to theFaceRecLib, but you should do that with care.
Simply use--to separate options to thebin/iet2014_face_recog.pyfrom options to theFaceRecLib.
For example, the--forceoption might be of interest.
See./bin/faceverify.py--helpfor a complete list of options.It is advisable to use the--dry-runoption before actually running the experiments, just to see that everything is correct.
Also, the Info (2) verbosity level prints useful information, e.g., by adding the--verbose--verbose(or shortly-vv) on the command line.
A commonly used command line sequence to execute the face recognition algorithm on both databases could be:Run the experiments on the MOBIO database:$ ./bin/iet2014_face_recog.py -vv --databases mobioRun the experiments on the SCface database, using protocol-specific files for the ZT-norm:$ ./bin/iet2014_face_recog.py -vv --databases scfaceRun the experiments on the SCface database, using files from all distance conditions for the ZT-norm:$ ./bin/iet2014_face_recog.py -vv --databases scface --combined-zt-norm --protocols close medium farNoteAll output directories of the scripts will be automatically generated if they do not exist yet.WarningThe execution of the script may take a long time and require large amounts of memory – especially on the MOBIO database.
Nevertheless, the scripts are set up such that they re-use all parts of the experiments as far as this is possible.Evaluating the experimentsAfter all experiments have finished successfully, the resulting score files can be evaluated.
For this, thebin/iec2014_evaluate.pyscript can be used to create the Tables 3, 4, 5 and 6 of thePaper, simply by writing LaTeX-compatible files that can later be interpreted to generate the tables.Generating output filesAlso, all information are written to console (when using the-vvvoption to enable debug information), including:The\(C^{\mathrm{min}}_{\mathrm{ver}}\)of the development set, the\(C^{\mathrm{min}}_{\mathrm{ver}}\)of the evaluation set and the\(C_{\mathrm{ver}}\)of the evaluation set based on the optimal threshold on the development set.The\(C_{\mathrm{frr}}\)on both development and evaluation set, using the threshold defined atFAR=1%of the development set.The\(C_{\mathrm{ver}}\)on the development and evaluation set, when applying threshold\(\theta_0=0\)(mainly useful for calibrated scores).The\(C_{\mathrm{cllr}}\)performance on the development and the evaluation set.The\(C^{\mathrm{min}}_{\mathrm{cllr}}\)performance on the development and the evaluation set.All these numbers are computed with and without ZT score normalization, and before and after score calibration.To run the script, some command line parameters can be specified, see./bin/iec2014_evaluate.py--help:--result-directory: Specify the directory where final result files are stored (default:results). This should be the same directory as passed to thebin/iec2014_execute.py`script.--databases: Specify a list of databases that you want evaluate. Possible values arescfaceandmobio. By default, both databases are evaluated.--protocols: Specify a list of protocols that you want to evaluate. Possible values arecombined,close,mediumandfarfor databasescface, andmaleandfemaleformobio. By default, all protocols are used.--combined-zt-norm: Evaluate the face recognition experiments on the SCface database with combined ZT-norm cohort.--combined-threshold: Evaluate the face recognition experiments on the SCface database by computing the threshold on the combined development set.--latex-directory: The directory, where the final score files will be placed into, by default this directory islatex.Again, the most usual way to compute the resulting tables could be:Evaluate experiments on MOBIO:$ bin/iet2014_evaluate.py -vvv --database mobioEvaluate experiments on SCface with distance-dependent ZT-norm:$ bin/iet2014_evaluate.py -vvv --database scfaceEvaluate experiments on SCface with distance-independent ZT-norm:$ bin/iet2014_evaluate.py -vvv --database scface --combined-zt-norm --protocols close medium farEvaluate experiments on SCface with distance-independent threshold (will mainly change the\(C_{\mathrm{ver}}\)of the evaluation set):$ bin/iet2014_evaluate.py -vvv --database scface --combined-threshold --protocols close medium farThe experiments to compare linear calibration with categorical calibration as given in Table 7 of thePaperare run using thebin/iet2014_categorical.pyscript:$ bin/iet2014_categorical.py -vvvGenerate the LaTeX tablesFinally, the LaTeX tables can be regenerated by defining the accordant\Resultand\ResultAtZeroLaTeX macros and include the resulting files.
E.g., to create Table 3 of thePaper, define:\newcommand\ResultIII[2]{\\}
\newcommand\ResultII[9]{#1\,\% \ResultIII}
\newcommand\Result[9]{#1\,\% & #4\,\% & #2\,\% & #3\,\% & #5\,\% & #6\,\% & #9\,\% & #7\,\% & #8\,\% &\ResultII}
\newcommand\ResultAtZero[8]{}set up yourtabularenvironment with 10 columns and input at according places:\input{latex/mobio_male}
\input{latex/mobio_female}
\input{latex/scface_close}
\input{latex/scface_medium}
\input{latex/scface_far}
\input{latex/scface_combined}Accordingly, the other tables can be generated from files:Table 4a):latex/scface_close-zt.tex,latex/scface_medium-zt.texandlatex/scface_far-zt.texTable 4b):latex/scface_close-thres.tex,latex/scface_medium-thres.texandlatex/scface_far-thres.texTables 5 and 6:latex/mobio_male.tex,latex/mobio_female.tex,latex/scface_close-zt.tex,latex/scface_medium-zt.tex,latex/scface_far-zt.texandlatex/scface_combined.tex.Table 7:latex/calibration-none.tex,latex/calibration-linear.texandlatex/calibration-categorical.texGenerate the score distribution plotsAt the end, also the score distribution plots that are shown in Figures 3 and 4 of thePapercan be regenerated.
These plots require the face recognition experiments to have finished, and also the categorical calibration to have run.
Afterwards, the scriptbin/iet2014_plot.pycan be executed.
Again, the script has a list of command line options:--result-directory: Specify the directory where final result files are stored (default:results). This should be the same directory as passed to thebin/iec2014_execute.py`script.--figure: Specify, which figure you want to create. Possible values are 3 and 4.--output-file: Specify the file, where the plots should be written to. By default, this isFigure_3.pdforFigure_4.pdffor--figure3or--figure4, respectively.Hence, running:$ ./bin/iet2014_plot.py -vv --figure 3
$ ./bin/iet2014_plot.py -vv --figure 4should be sufficient to generate the plots.