ElisonSherton/github-issues huggingface dataset:
GitHub issue: Errors when running install_packages.sh
State: open

Hi,

we are preparing a NIST submission and are facing some issues with the setup of our build & validation environments. In order to make sure we are including all required files in `/lib` for dynamic linking we 
- building our library and exporting the output folders (eg: `lib`, `config` and `doc`) in a `development` image
- copying these output folders only into a `validation` image and running the validation scripts there

This validation image is just a raw `focal` image where we have run
- apt update
- apt -y install sudo lsb-release

And then we run the `install_packages.sh` script. Problem is the script fails to install many of the packages seemingly due to missing versions on the package repo. For instance **_libavcodec-dev_** install fails with
`E: Version '7:4.2.4-1ubuntu0.1' for 'libavcodec-dev' was not found`

If we edit the **_install_packages.sh_** script removing the pinned versions then everything seems to work correctly, though we would no longer be sure to be running in the exact same environment as the NIST benchmark.

Is it possible that the **_install_packages.sh_** is pointing to versions no longer available on the public repo and thus needs to be updated, or am I missing something?

Thanks in advance,
Filipe Morais

Thanks for bringing this to our attention.  We will plan to include the actual packages themselves along with an updated install script to address this issue (hopefully in the next week or so).

[ubuntu iso copy provided at this repo is being used.](https://nigos.nist.gov/evaluations/ubuntu-20.04.3-live-server-amd64.iso)

I have the same problem specifically the GCC compiler.

The documentation states that GCC version is 9.3.0

while it is 9.4.0 after running install_packages.sh 

There are multiple packages that have the same issue.

Should I wait for modification? or package my library and submit with the default installed library?

I  appreciate your help @mlngan 




Please go ahead and run/validate with the default installed libraries.  The original approach to fixing install_packages.sh is getting a bit more complicated than we originally thought.  We will keep this issue open and try to come up with a reasonable solution.
GitHub issue: NIST FRVT version number symbols
State: closed

According to `frvt_structs.h` I must define NIST FRVT version number symbols in the core library. 

https://github.com/usnistgov/frvt/blob/7c2e52155467f0858cb30c9c688cb47e56142366/common/src/include/frvt_structs.h#L502-L519

What should the values be?

I am basing my implementation on the [nullimplfrvtpad.cpp](https://github.com/usnistgov/frvt/blob/master/pad/src/nullImpl/nullimplfrvtpad.cpp), but I don't see it defined in the source code.

There's nothing you have to do explicitly.  Those version number symbols are defined in ./pad/src/testdriver/CMakeLists.txt

set (CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wall -std=c++17 -DNIST_EXTERN_FRVT_STRUCTS_VERSION -DNIST_EXTERN_API_VERSION")
GitHub issue: Pose Orientation at Test Time FRVT 1-1
State: closed

In validation package, some images are orientated 90 degrees from the front, as such sometimes we are comparing frontal faces against side faces OR side faces against side faces.

May I know would similar facial orientation (frontal against side OR side against side) be used during testing and evaluation by NIST?

Some of our unconstrained, non-cooperative datasets can contain images with large pose variations.
GitHub issue: Template Creation Failed Situation and Its Evaluation
State: open

Hi @mlngan ,

I wonder what is the evaluation criteria if I encounter with template creation failed situation. More precisely, suppose I couldn't create a template for a specific image, then filled its vector to handle it in my matcher. And, for example, I return 0 (zero)  similarity score for that template in my matcher. Then in the evaluation stage, will Nist evaluate that template for FNMR / FMR calculations? I mean, will Nist take "that failed template-based image" into consideration in the evaluation stage or not? I'm asking this question because if that template will be taken into consideration, I shouldn't give a negative score in my matcher stage.
GitHub issue: Distribution of actual matches during validation 
State: closed

Hello, I have successfully ran the validation, and it seems that vast majority of matching has very high similarity scores, except for only 1. May I know if that is the norm for the validation package? Are these comparisons suppose to be matches? What should be the distribution of matches in the validation?

Thank you!

This is heavily implementation dependent.
In the validation package, some of the pairs are incorrect (you can tell because the IDs are different). A match is expected when the ID is the same.

Here's an example output with 'distance' (opposite of similarity) vs expected match for a solution that gets 100% in validation
[v8.txt](https://github.com/usnistgov/frvt/files/9920710/v8.txt)
GitHub issue: how to pass an array of array from python to C++ (multiple faces in an image) 
State: closed

I am doing template generation for multiple faces in image:

I am returning an array of array (array of templates) from python to c++. Here is a snippet of my codes:

```
npy_intp dims[3] = {height,width,channel};
np_arg = reinterpret_cast<PyArrayObject*>(PyArray_SimpleNewFromData(3,dims,NPY_UINT8,image.data.get()));
pArgs = PyTuple_New(1);
PyTuple_SetItem(pArgs, 0, reinterpret_cast<PyObject*>(np_arg));
pValue = PyObject_CallObject(pFunc, pArgs);
if  (pValue==nullptr){
        PyErr_Print();
}
np_ret = reinterpret_cast<PyArrayObject*>(pValue);
c_out = reinterpret_cast<float**>(PyArray_DATA(np_ret));
std::cout << c_out[0][0] << std::endl;
```

During runtime, i encounter segmentation error at std::cout << c_out[0][0] << std::endl; (PID 8132 exited due to signal 11)
std::cout << c_out[0] << std::endl; runs fine though.

What did I do wrong?

If you dereference the pointer and there's nothing there, it'll crash. That appears to be the case.
For example, if c_out[0] is nullptr (0), what do you expect?
I see you print something else and then continue to the crash.

I'm not sure what the Py* functions are doing but it appears like you have a float* rather than float** and are trying to cast it incorrectly.

These questions do not belong here. They aren't related to FRVT.
GitHub issue: Can you provide minimal docker image?
State: closed

Hi, it will be great if you can provide minimal docker image for running validation scripts. Thanks.

The base image seems to be the same you get from the official ubuntu repo for 20.04.3, so I think it should be okay to use a corresponding base dockerfile such as `ubuntu:focal-20220113`. Still would be preferable to have an official response from NIST for sure 🤞

Per the our published documentation, the operating system that the submitted implementations shall run on is released as a downloadable file accessible from https://nigos.nist.gov/evaluations/ubuntu-20.04.3-live-server-amd64.iso, which indeed is the 64-bit version of Ubuntu 20.04.3 LTS (Focal Fossa).

This is an older issue but I think it is still relevant. Using an iso file is much less convenient than using a docker image in CI/CD systems like jenkins, that are frequently used to automatically build and test releases. 
Given that it the OS requirement is quite specific and not the most recent LTS release of Ubuntu 20, it is quite hard to find something that works. 

I use [this](https://hub.docker.com/r/swathip2021/myubuntu) image in the first line of my Dockerfile.
```
FROM swathip2021/myubuntu:20.04 AS ubuntu
```
GitHub issue: Necessary to output eye coordinates?
State: closed

I am working on FRVT 1:1, and my query is : is it necessary to output eye coordinates? I mean the goal is to compare templates between template for enrollment and verification. 

Can I leave the eye coordinates as zero?



Yes, you don't have to populate the eye coordinates.

May I also ask: I understand that during test, images contain one and only one face. But in the codes (for multiple faces in an image) in a given in nullImplfrvt11.cpp:

```
NullImplFRVT11::createTemplate(
    const FRVT::Image &image,
    FRVT::TemplateRole role,
    std::vector<std::vector<uint8_t>> &templs,
    std::vector<FRVT::EyePair> &eyeCoordinates)
{
    int numFaces = rand() % 4 + 1;
    for (int i = 1; i <= numFaces; i++) {
        std::vector<uint8_t> templ;
        /* Note: example code, potentially not portable across machines. */
        std::vector<float> fv = {1.0, 2.0, 8.88, 765.88989};
        /* Multiply vector values by scalar */
        for_each(fv.begin(), fv.end(), [i](float &f){ f *= i; });
        const uint8_t* bytes = reinterpret_cast<const uint8_t*>(fv.data());
        int dataSize = sizeof(float) * fv.size();
        templ.resize(dataSize);
        memcpy(templ.data(), bytes, dataSize);
        templs.push_back(templ);

        eyeCoordinates.push_back(EyePair(true, true, i, i, i+1, i+1));
    } 
    
    return ReturnStatus(ReturnCode::Success);
}
```
Is it necessary to return multiple templates for a given image of more than one faces? Can I not populate this? Or perhaps return only the template of the largest face?

Appreciate the clarification :)




@tanpengshi You should read the requirements here https://pages.nist.gov/frvt/api/FRVT_ongoing_11_api.pdf. 
There are 2 templates: the first is a template with input Multiface (each image contains only a face and the same face of a person) You can return the largest face if your detector detects the number of faces > 1. The second is a template with input only an image (can contain one or more faces), you must return all faces for that.

Can we also assume images at test time are either greyscale (1 channel) or RGB channels?

Images used in the evaluation will either be 8-bit grayscale (1 channel) or 24-bit (3 channels) color.
GitHub issue: Query on time limit during testing for 1-1
State: closed

I am embedding my Python codes in C++.

My concern is regarding the time taken for each image inference, because I understand the time limit for image enrollment is 1500ms. Using this code above i take about 2.5seconds to 3 seconds (on Virtual Machine CPU) to iterate between each std::cout during runtime, as shown:

![image](https://user-images.githubusercontent.com/24505990/198968574-729f0fd2-7a62-49cc-99b2-10f1eba88921.png)

However, during testing with Python on my CPU work station, each template generation only takes about 190ms, as shown:

![image](https://user-images.githubusercontent.com/24505990/198969045-7465677d-b70b-4ecb-88a6-771f37c03dcc.png)

My question is how are the codes being tested for timings, and would my codes potentially exceed the time limit? Appreciate the help!







It is 1.5 second per feature generation (detection + recognition) on a single core of the CPUs specified in the API document.

In your Python example, it is running multi-threaded and looks like you are only doing face recognition on a 112x112 image, which is a far easier task.
GitHub issue: [Error] Enrollment validation (multi person in image) failed - when no face detected
State: closed

on function createMultiTemplates() in validate11.cpp, i found that it requires "the output template vector must contain at least one template"; but in some situations, model face-detector cannot detect anything for example, we don't return any templates and that cause this error. 
So can anyone help me what i should return at this case or any solution to solve this problem?

Per the [FRVT 1:1 API](https://pages.nist.gov/frvt/api/FRVT_ongoing_11_api.pdf), if the implementation is unable to extract features, the output shall still contain a single template that may be passed to the matchTemplates() function without error. That is, this routine must internally encode "template creation failed", and the matcher must transparently handle this.  A template may have length zero bytes as long as your matcher handles it.
GitHub issue: Error Detection of hard-coded config directory  when run ./run_validate_11.sh 
State: closed

Running ./run_validate_11.sh shows me the following error:

**Checking installation of required packages [SUCCESS]
Looking for core implementation library in /home/daksha/Documentos/frvt/11/lib.[SUCCESS] Found core implementation library /home/daksha/Documentos/frvt/11/lib/libfrvt_11_facedaksha_000.so.
Attempting to compile and link /home/daksha/Documentos/frvt/11/lib/libfrvt_11_facedaksha_000.so against test harness.
Scanning dependencies of target validate11
[ 33%] Building CXX object src/testdriver/CMakeFiles/validate11.dir/home/daksha/Documentos/frvt/common/src/util/util.cpp.o
[ 66%] Building CXX object src/testdriver/CMakeFiles/validate11.dir/validate11.cpp.o
[100%] Linking CXX executable ../../../bin/validate11
[100%] Built target validate11
[SUCCESS] Built executable in /home/daksha/Documentos/frvt/11/bin.
------------------------------
 Running 1:1 validation
------------------------------**
Checking for hard-coded config directory PID 52211 exited due to signal 11
[ERROR] Detection of hard-coded config directory in your software.  Please fix!

I do not hard-coded config in source code, create config folder for my models.

Your program threw a Segmentation Fault error (signal 11). The `run_validate_11.sh` script just checks whether the programs return a successful exit code 0.

thanks

@jatsi  did you solve it?
GitHub issue: thread count and mxnet forks
State: open

We are using mxnet for our algorithm, we are able to reduce the number of threads to 5 with only 1 running thread, however to remove display messages we have built the library from source by commenting the logs where we are facing issues with non returning forks. The code executes for the images involved in one phase for example enrolment and then it waits for the process to return. 
Can anyone help me to resolve it?
GitHub issue: Query regarding the Report on demographic groups
State: open

Hello

We have a few queries regarding the upcoming "Report on how error rate variations across demographic groups." 

1. Will this report be based only on the algorithms submitted to FRVT 1:1 leaderboard, or will this report also contain an analysis of algorithms submitted to the FRVT 1:N leaderboard? 
2. Will the new algorithms submitted to FRVT 1:1 and FRVT 1:N leaderboards next week also be a part of this demographics report?
3. What will be the frequency of this report? Will this be released monthly like the FRVT 1:1 and FRVT 1:N reports?

Thank you!
GitHub issue: Query on the updated protocol that supports the existence of multiple faces detected in an image
State: closed

Hello

In the recently released [FRVT 1:1 leaderboard](https://pages.nist.gov/frvt/html/frvt11.html) along with benchmark numbers (in blue) with the updated protocol that supports the existence of multiple faces detected in an image, the ranks and the FNMR numbers (in blue) on the `BORDER Photos` and `KIOSK Photos` benchmark sets are extremely large compared to other benchmarks where this updated protocol isn't used even for the algorithms which are very well ranked on the other benchmark sets.

Could you kindly give us more clarity on how these FNMR values reported in blue on the leaderboard should be read? 
And are these high FNMR rates in blue with updated protocol due to inaccuracies in the submitted algorithms, or should we take into account anything else from the package and benchmark data point of view? 

I really appreciate any help you can provide. Thank you.

In the updated protocol that supports multiple faces detected in an image, we take the maximum similarity score across all template comparisons between each enrollment and verification image pair.  The high ranks (poor FNMR) are due to higher FMR under the max(score) policy.  The upside to this approach is of reduced FNMR – this can occur because in a genuine comparison of say an image with 1 face present with another image with 2 faces present, detection of the correct mated person was not guaranteed before we changed the API.  With the new API, one of the two faces should score highly.  We take largest score over those produced in comparing K1 templates from image 1 with K2 templates from image 2, i.e. max of K1 * K2 scores.   We do this for mated **and** non-mated comparisons.
GitHub issue: If my code needs to read the model externally, where should I put the face recognition related models ？
State: closed

Take 1:N as an example. In the NullImplFRVT1N::initializeTemplateCreation  func,  I  should load the face recognition related models. If puting the face recognition related models in the same level directory as run_validate_1N.sh,  may i use './face_model' in the initializeTemplateCreation function?

```
NullImplFRVT1N::initializeTemplateCreation(
    const std::string &configDir,
    FRVT::TemplateRole role)
{
    face_model model = load('./face_model');
    return ReturnStatus(ReturnCode::Success);
}
```
the following is the tree struct of 1N directory. 
|-- CMakeLists.txt
|-- README.md
|-- bin
|   `-- validate1N

|-- config
|-- doc
|   `-- version.txt

|-- input
|   |-- enroll_1N.txt
|   |-- enroll_1N_multiface.txt
|   |-- enroll_1N_short.txt
|   |-- insert.txt
|   |-- search_1N.txt
|   `-- search_1N_multiperson.txt

|-- lib
|   `-- libfrvt_1N_xxx_000.so

|-- libfrvt_1N_xxx_000.tar.gz
|-- run_validate_1N.sh
|-- scripts
|   |-- build_null_impl.sh
|   |-- compile_and_link.sh
|   `-- run_testdriver.sh

`-- validation


The models should be located in the 'config' directory.

I don't think there's any guarantee over the other directories which is why the configDir is supplied in the function.

> The models should be located in the 'config' directory.
> 
> I don't think there's any guarantee over the other directories which is why the configDir is supplied in the function.

thanks.
GitHub issue: match against failed templates
State: closed

Hi,

for frvt_11, I read in th documentation of the template creation:
```
In all cases, even when unable to extract features, the output shall be a template that may be
passed to the matchTemplates() function without error. That is, this routine must internally encode "template
creation failed" and the matcher must transparently handle this.
```

Fine, but what about similarity?
Evene if I return some error during template creation step, that template is still being passed to `matchTemplates` and returning a similarity makes no sense.
The failed templates shouldn't be supposed to not being passed to match functions? If not, what similarity should I return in those cases?

Thanks

You can set the similarity score to whatever value you think is a good idea, for example, returning a very low score to represent a rejection or setting the similarity score to -1.0.
GitHub issue: Tensorflow is not fork safe
State: closed

Hi NIST team.

I've created a dynamic library from Rust that my implementation of "frvt.h" in c++ uses, but I have a problem.
My model inference is made with TensorFlow, when I run the "run_validate.sh". It hangs indefinitely.
I've read that TF is not fork safe so that might be the problem. 

¿Do you suggest to try another Machine Learning framekork or is there a workaround for this issue?

Edit: added a missing word "framework".


All the main frameworks such as tensorflow, pytorch and mxnet have been known to work in NIST.
You may just need to compile it differently or set it up (usually a setting in each framework) so it does not attempt to use multi-threading.

Okay, I'll see if it is a multi thread problem and if I can recompile it without multi-threading.
Also, I'm closing this issue for now.

Hi NIST team.
I've searching a lot and sadly, I've had the same problem. 
I even migrated my TF Rust code to CPP with no luck.

Researching a little more I foud an open issue regarding NIST implementatinos in the official TF github and they say as follows:
[https://github.com/tensorflow/tensorflow/issues/51832#issuecomment-926205545
](https://github.com/tensorflow/tensorflow/issues/51832#issuecomment-926205545)
So basically NIST API is not compatible with all model inferences with TF and the TF developers have not intenssions to make TF fork safe.

¿Do you have any plans to make more compatible your API with TF?
For me it's contradictory that you use forks when TF, one of the more spreaded IA framekors, is not fork safe.

I've seen that TF allows "sessions" as servers, and "session-clients". But that will require an extra process to use it. I assume that that implementation is not allowed by NIST.

We've received a number of submissions using TF.  It's possible to move your model initialization to the child process (after the fork) and invoke it once during the first API call.  As for measuring timing, we consider and report the median time across thousands of API calls (e.g. createTemplate()), so even if your very first API function invocation takes longer, it should not impact the timing benchmark.

Thanks. That was my first approach, but TF in Rust takes a long time to initialize. In c++ the initialization is faster.
Now  I'm initializing at "matchTemplate" once per fork.

Just as conclussion for this issue. If you initialize your TF model at "initialize()"w and it hangs at "matchTemlate", you should try to initialize it once per fork at "matchTemplate" or "createTemplate". Regarding the time, it may not affect considerably the benchmark due the mean calculatiuon of thousands of API calls.

¿Am I right?
GitHub issue: Does the implementation of FRVT 1:N search also need to be limited to a single process？ 
State: closed

As shown in the following picture, you need to test the speed of 1:N search with one single-core, but in the run_testdriver.sh script, only the number of processes during createTemplate is counted, but not during 1;N search.  Does the implementation of FRVT 1:N search also need to be limited to a single process？ 

![image](https://user-images.githubusercontent.com/24365567/153814230-b302548e-8708-41fa-9f0e-649698e14b63.png)


Yes, the implementation of 1:N search needs to be limited to a single process.
GitHub issue: input/match_multiperson.txt does not exists?
State: closed

```
echo -n "Matching Multi-person Templates (Single Process) "
inputFileMulti=input/match_multiperson.txt
rm -f $inputFileMulti
sed '1d' $inputFile | awk '{ print $1 }' | while read line
```

https://github.com/usnistgov/frvt/blob/ubuntu/11/scripts/run_testdriver.sh#:~:text=inputFileMulti%3Dinput/match_multiperson.txt




The match_multiperson.txt generated from the following code;
```
do
    grep "${line}_" $outputDir/$outputStem.log | awk -v enroll=$line '{ print enroll".template " $1".template" }' >> $inputFileMulti
done
```
GitHub issue: ubuntu 20.04.3 LTS evaluation gcc version
State: closed

Should gcc version be 9.3 (ubuntu 20.04.3 LTS default) or 8.3.1 as usual, when switching the FRVT evaluation operating system to Ubuntu 20.04.3 LTS?
Thx

I used gcc 11 with CentOS (via epel) on previous submissions.
Ubuntu 20.04 comes with gcc-10 in the standard repo by the way.

The version of gcc is unlikely to be the matter but rather the version of glibc, which any compiler on Ubuntu 20.04 should use.

Our Ubuntu image is running g++ version 9.3.0.
GitHub issue: match_multiperson.txt file is missing.
State: closed

The last test in ./run_validate_11.sh regarding "Matching Multi-person Templates (Single Process) " needs match_multiperson.txt which is not available in the input folder.
Please advise on how to proceed. 

If your software successfully runs through validation software, match_multiperson.txt gets generated at the end in scripts/run_testdriver.sh.
GitHub issue: Thread count
State: closed

I have limited thread of Tensorflow and Pytorch to 1, but still the warning message mentions about detecting 6 threads. Out of these 6 threads only one thread is running and others are in sleep mode. The following is the output of htop command for the above mentioned scenario. In one of the comment it is mentioned that you check multithreading case by case, Is this case acceptable for submission?
![Screenshot (537) - Copy](https://user-images.githubusercontent.com/25266500/133846525-ad5b2849-f2e9-4389-b67e-1a6f4b5ad2ea.png)


We are only interested in the active threads, so given you only have a single active thread, it should be fine.  Send us your submission, and we will validate the active thread count on our end.  Thanks.
GitHub issue: Resolution constraints for the input image?
State: closed

What is the minimum and maximum resolution of input image to createTemplate method  or is it 640x480? 

createTemplate(
const Multiface &faces,
TemplateRole role, Input
std::vector<uint8_t> &templ,
std::vector<EyePair> &eyeCoordinates);

There is no minimum/maximum resolution per se.  In our report and on our leaderboard under the resources tab, you'll see that we run on images as large as 3000x4500, sometimes larger.  If your software inherently does not support certain image sizes, it should fail gracefully, meaning return a successful return code and encode the failure within your template.  It should not crash.
GitHub issue: Repo takes a very long time to clone 
State: closed

Due to all the image files which are stored in the github repo, the repo takes a **very** long time to clone.
We build our NIST FRVT implementation as part of a CI pipeline, and it is currently taking over 10 minutes to clone the repository. 

Is it possible to store the images outside of the github repo on a faster data storage platform (ex. google cloud storage), and perhaps provide a shell script to download and extract the images to the correct location? 

I was able to reduce the build time significantly by caching the github repo.
GitHub issue: galleryInsertID in 1N
State: closed

Should I change the implementation of galleryInsertID(const std::vector<uint8_t> &templ, const std::string &id) function frvt/1N ? What do I loose if keep the implementation of this function exactly as that of frvt/1N/src/nullImpl/nullimplfrvt1N.cpp ? 

The null implementation currently stores templates in a std::map, as an example.  If your algorithm stores templates with a different data structure (e.g., to enable faster search), then what's in the null implementation is obviously not going work.
GitHub issue: Libarchive and cmake version mismatch   
State: closed

While running the scripts provided by NIST to compile and build the final package we are getting an error 

`cmake: symbol lookup error: cmake: undefined symbol: archive_write_add_filter_zstd`

In order to resolve the issue we had to upgrade the libarchive to 3.3.3. Also, the issue was not seen with CMake version 3.17. 

Question: Is this issue been already resolved from NIST's end, since the repo is quite old and maybe has not been updated.

Should we go ahead with a version that works or it has to be the version in NIST's scripts install?

[Similar issue ](https://bugzilla.redhat.com/show_bug.cgi?id=1827927)

Interesting.  We've never encountered this issue with cmake on our end.  Our systems are running CentOS Linux release 8.2.2004, cmake version 3.11.4, and libarchive.so.13.3.2.

We are seeing this error with CMake version 3.18. So I think cmake 3.11.4 wont have issue, we can submit the package.

For cmake 3.11 libarchive is not required , but for 3.18 which your script installs libarchive might be required, so do u think the repo script should be updated?

I just did a fresh install of the CentOS 8.2 image on a VM and the latest cmake package being pulled from AppStream is indeed version 3.18.2-11.  I was able to replicate your issue with libarchive, so I've added the libarchive package to the installation list in the validation scripts (which should pull the correct/compatible version).

I've just fresh install the CentOS 8.2 on VM, and even the new commit has added libarchive but the issue is still.
It seems that libarchive 3.3.2 is already installed in my fresh installed CentOS. I solved it by update libarchive to 3.3.3 using:
`sudo yum update libarchive`

Thanks for making us aware of this.  We have added the libarchive update to the validation scripts.
GitHub issue: Testing the final submission package
State: closed

Hello

For example in 1:N package, upon successfully running `run_validate_1N.sh` script (in a fresh CentOS 8.2 system) without any error, it will create a submission package compressed in tar.gz format. How can we use this generated package to further test whether this submission package will run from NIST's side or not?

If we are able to prepare the submission package without any error, but during validation from NIST team, some errors pop up, how can we replicate these errors from our side? Before submitting the encrypted package, is there any script/command that we can use to test the validity of the submission package (the tar.gz file generated) like the NIST team would do?

Thanks in advance.

If the validation script successfully ran and was able to generate the submission package, then that means it passed all preliminary validation checks, and you can just send us the encrypted tar.gz package to NIST.

If there are errors during validation on our end, we will notify you with the issues.
GitHub issue: What happens if the submission fails timing tests?
State: closed

Hello,

If the package we submitted runs successfully without package/code/threads errors, but it does not pass the timing tests (Eg: Under 1500 ms for template creation in 1:1 package). Would we get a chance to fix the package and resend it immediately or would we have to wait for 4 calendar months?

If we can fix and resend the package immediately, will the submission be evaluated in the current iteration leaderboard update or the next update iteration?

Thanks in advance.

If your submission fails timing, you will have one opportunity to resubmit immediately to resolve the timing issue.  We process submissions on a first-come-first-serve basis, so once we receive a working submission from you, we will run it and include it in the next report cycle.
GitHub issue: 1:N submission - How to return templates when faces are not detected?
State: closed

In cases where faces are not detected by the detector, 
1) should we return an error code or a random template (feature vector, eye coordinates)? 
2) How will these templates be handled during enrolment, search phase?

Per the API document (https://pages.nist.gov/frvt/api/FRVT_ongoing_1N_api.pdf):

For enrollment templates (TemplateRole=Enrollment_1N): When the implementation fails to produce a template (i.e. returns a non-successful return code), it shall still return a blank template (which can be zero bytes in length). The template will be included in the enrollment database/manifest like all other enrollment templates, but is not expected to contain any feature information.

For identification/probe templates (TemplateRole=Search_1N): If the function returns a non-successful return status, the output template will not be used in subsequent search operations.
GitHub issue: Timing analysis
State: closed

Hello,

Do you use a separate script to perform timing analysis?
Would it be possible for you to share the code you use to test template creation time?
Thanks.

We are basically doing the following (pseudocode):

start = std::chrono::system_clock::now()
createTemplate()
stop = std::chrono::system_clock::now()
time = stop - start


@mlngan Thanks for the reply.
But does this code use any forking?
In your comment here https://github.com/usnistgov/frvt/issues/12#issuecomment-501275609 you mentioned "we lock all threads to using a single CPU".  Does this mean that the validation package is tested on a multi-cpu system, but only confined to single cpu via code? 



I'm sure they just set the process affinity using taskset or similar.
taskset -c$assignedcore ./testVendor

It is tested on one of the machines they list in the General Evaluation document. 
 Dual Intel® Xeon® E5-2630 v4 CPU @ 2.20GHz (10 cores each)
 Dual Intel® Xeon® E5-2680 v4 CPU @ 2.4GHz (14 cores each)
 Dual Intel® Xeon® Gold 6140 CPU @ 2.30GHz (18 cores each)
GitHub issue: Threshold to make the decision in FRVT 1:N package
State: closed

In Table 10 on page 13 of [FRVT 1:N API document](https://pages.nist.gov/frvt/api/FRVT_ongoing_1N_api.pdf), we are asked to return a `decision` (true/false) whether there is a mate within the enrollment database for a query along with the candidate IDs and similarity score. What threshold should be used to make this decision in the [`identifyTemplate`](https://github.com/usnistgov/frvt/blob/master/1N/src/nullImpl/nullimplfrvt1N.cpp#L106) function?

On the [1:N leader board page](https://pages.nist.gov/frvt/html/frvt1N.html), it is told that `"threshold is set to limit to the False Positive Identification Rate (FPIR) to 0.003"` and also that `"The threshold is set for each algorithm and each column separately"`. So, we are assuming NIST would be setting the thresholds to get the final results.

So, we are confused on which threshold to use to make the decision in [`identifyTemplate`](https://github.com/usnistgov/frvt/blob/master/1N/src/nullImpl/nullimplfrvt1N.cpp#L106) function. Any help with this will be much appreciated.

Thanks in advance. 

The **decision** would a best guess.  You might apply a threshold, or you might do something else (for example, what if more than one person comes above threshold in a consolidated database?).  We'd use the **decision** to compute one point on a Detection Error Tradeoff (DET) curve.  It would probably lie on the DET we report (which sweeps across a number of different thresholds).

We're not going to use the **decision** for algorithm comparison.
GitHub issue: Script for FRVT 1:1 testing of multiface input 
State: closed

Hi,
According to your documentations, a person can be represented by multiple images (and these images will be represented as a single std::vector<Image> . However your in your test scripts for FRVT 1:1, you only run test for one single enroll image - one verification image. Is there any way so we can test multiple enroll image - one/multiple verification image ? 


We've added a test case in the validation package for template generation using multiple image inputs.
GitHub issue: Range of similarity 
State: closed

Hello,

According to the document, the similarity value is a similarity score resulting from comparison of the templates, **on the range [0,
1000000].**
We use cosine similarity to calculate the similarity score, the value is on the range [0,1].
Do we need to normalize that value from [0,1] to [0, 1000000]?

Thanks,

No.  Any non-negative range is permitted eg. [0,1] or [300,400].


From: Kei ***@***.***>
Date: Tuesday, May 18, 2021 at 22:50
To: usnistgov/frvt ***@***.***>
Cc: Subscribed ***@***.***>
Subject: [usnistgov/frvt] Range of similarity (#52)

Hello,

According to the document, the similarity value is a similarity score resulting from comparison of the templates, on the range [0,
1000000].
We use cosine similarity to calculate the similarity score, the value is on the range [0,1].
Do we need to normalize that value from [0,1] to [0, 1000000]?

Thanks,

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub<https://github.com/usnistgov/frvt/issues/52>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ADVZFNILBRDYXBK6X3U6BUTTOMRPTANCNFSM45DWHBEA>.
GitHub issue: Testing Limits
State: closed

I've got that submission has to use 1 CPU thread. What are the other limits: inference time, RAM usage, etc?...

The timing limits for 1:1 are listed in Table 1 of the API document: https://pages.nist.gov/frvt/api/FRVT_ongoing_11_api.pdf.

Please see the FRVT Evaluation Specifications for hardware specs: https://pages.nist.gov/frvt/api/FRVT_common.pdf.
GitHub issue: Can I load the models in the constructor? 
State: closed

Hello, 

When I execute run_validate.sh, it does not seem to be calling the initialize function. I don't see the messages I print nor a change of the values of the variables. Would it be a problem if I loaded the model files within the constructor?

The initialize() function is called in the testdriver provided in the validation package - https://github.com/usnistgov/frvt/blob/master/11/src/testdriver/validate11.cpp

Did you follow the instructions contained in the README.md file on putting your library into ./lib?  Or did you optionally build the null sdk?  Did you successfully compile and link your library against the provided testdriver - e.g. did you see any errors when you ran run_validate_11.sh?  Additionally, the path to the configuration directory that you will read your models from is passed in as a parameter to initialize().  Please do not hard-code your configuration directory, because it will not be in the same location on our systems.

It works when I build the null sdk.
GitHub issue: How to confirm which pictures are the same person
State: closed

Hi @mlngan how do i categorize the pictures provided on github(https://github.com/usnistgov/frvt/tree/master/common/images)?
How do i know which pictures are the same person?

The purpose of the validation package is to ensure your submission adheres to the API and tests a bunch of edge cases to make sure your software doesn't crash.  The images published aren't necessarily used to test the accuracy of your software, so we do not publish any sort of ground truth for the identities.
GitHub issue: Waiting for 1:1 results..
State: closed

Its been more than 10 days and we haven't received results for 1:1 can you please give some clarity as you have moved to a different OS and we would have to migrate for submitting 1:N submission. Very confused. Please update.

Email us:  frvt@nist.gov
GitHub issue: Incorrect path
State: closed

When the [script]( https://github.com/usnistgov/frvt/blob/master/11/scripts/run_testdriver.sh
) is running, an empty path is passed.
The system centos 7.6 did not repeat itself.

```bash
echo -n "Checking for hard-coded config directory "
tempConfigDir=$(pwd)/otherConfig
chmod 775 $configDir; mv $configDir $tempConfigDir; chmod 550 $tempConfigDir
bin/validate11 createTemplate -x enroll -c $tempConfigDir -o $outputDir -h $outputStem -i $inputFile -t $numForks -j $templatesDir
retEnroll=$?
```

@mlngan 

Unsure why tempConfigDir would be an empty path as it works on my end.  But I've reverted back to not appending $(pwd) in front of the config dirs, so it should work now.  New OS requirement is now CentOS 8.2.  b18c70f
GitHub issue: update scripts for a centos 8.2
State: closed

When will the scripts be updated for Centos 8.2? @mlngan 


We'll be pushing out updates for CentOS 8.2 this week.
GitHub issue: Code in Python
State: closed

In order to submit to NIST FRVT, do we need to transfer our code from Python and TensorFlow to C++? Is it possible to call and execute the Python code from within the NIST FRVT C++ API?

The NIST FRVT API would require that you either translate or wrap your Python code with C++.  We have received submissions where developers have successfully done this.
GitHub issue: Twice call to distructor ~NullImplFRVT11()
State: closed

The NIST API calls ~NullImplFRVT11() two times after each call to "bin/validate11 createTemplate -x enroll -c $configDir -o $outputDir -h $outputStem -i $inputFile -t $numForks -j $templatesDir retEnroll=$?" Both the destructors use different process ids. This does not allow my program to release memory. I expect only one call to ~NullImplFRVT11(). Whats the problem and how to restrict processes to only one?

The testdriver forks child processes to do work, and the destructor gets called once for each child process + once for the parent process.  We use this approach in our actual test, so you will need to address this.
GitHub issue: Two submissions before the four months
State: closed

Hello, 

This is not really an issue, it is more a doubt. I believe last year it was possible to submit two versions of algorithms, and then every submission would need to wait for four months until a new one is done. I did it in the past company, so I'm sure about this.

Is this rule currently valid?

Thanks for the clarification.

Best,
Miguel

For the FRVT 1:1, 1:N, and Quality tracks, participants may send **one** submission as often as every four calendar months from the last submission for evaluation.  We are no longer allowing two submissions for first-time participants.


Thanks for the quick updade. Appreciate it!

@mlngan What if my submission has an error/problem in code, for example, generating too many threads. Do you still start counting 4 month for my submission? Or you start counting 4 month when my submission is running successfully on NIST's end?

You will be allowed to submit your next submission 4 months from the date you send us a successfully running algorithm.
GitHub issue: Error authenticate the Nist key
State: closed

I am confusing at email my public key to Nist, Don't really understand here. What is it and how do I know my public key is sent to Nist successfully?

![image](https://user-images.githubusercontent.com/19263564/88777491-79bbf880-d1b1-11ea-8c39-927a88291729.png)


Your public encryption key is the result of step 2 of the process, which should be named forfrvt11.gpg.  You need to email that file to frvt@nist.gov.

I need to re-submit my solution to Nist. But unfortunately, my default key (private key) disappears. Can I generate another key and resubmit, including forfrvt11.gpg? It is definitely different from the one printed in the agreement ( already sent by post).
Thank you

If you generate a new key, you will have to send us an updated participation agreement with the new key fingerprint and email us the new gpg file.
GitHub issue: Matching Templates (Multiple Processes) PID 23985 exited due to signal 11...
State: closed

I'm now testing frvt 1:1 for our solution.

I met the following messages:


Matching Templates (Multiple Processes) PID 23985 exited due to signal 11
PID 23986 exited due to signal 11
PID 23988 exited due to signal 11
[ERROR] Match validation failed


Can you guide how to fix the problem?

Thanks.

Hong.


Your messages are an indication that a segmentation fault occurred - you could try running valgrind or gdb to see what's causing the issue.  Perhaps also check that you're not releasing memory twice somewhere (e.g., in your destructor).
GitHub issue: confusing about fork()
State: closed

sorry for bothering you again @mlngan .
My face_detection model does not execute in the child process (no bug, no return anything) and the parent process is suspended( "wait" function in parent process is still waiting for return status from the child ). Please help.

solved: my software runs on multithread, That's why the system has frozen.
GitHub issue: Can I build a static shared library in linux debian and run it in CentOS ?
State: closed

Hello, I am trying to build -static shared library in ubuntu 18.04 so that it is able to run in CentOs but I got some "undefined reference to" errors when compiling "compile_and_run.sh". Does anyone try this solution? any comment is appreciated, thank you.
Scanning dependencies of target validate11
[ 33%] Building CXX object src/testdriver/CMakeFiles/validate11.dir/home/cybercore/workspace/frvt/common/src/util/util.cpp.o
[ 66%] Building CXX object src/testdriver/CMakeFiles/validate11.dir/validate11.cpp.o
[100%] Linking CXX executable ../../../bin/validate11
../../../lib/libfrvt_11_cybercore_001.so: undefined reference to `std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::compare(char const*) const'
../../../lib/libfrvt_11_cybercore_001.so: undefined reference to `std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_create(unsigned long&, unsigned long)'
../../../lib/libfrvt_11_cybercore_001.so: undefined reference to `__cxa_throw_bad_array_new_length'
../../../lib/libfrvt_11_cybercore_001.so: undefined reference to `operator delete(void*, unsigned long)'


@mlngan pls help. I know that the " undefined reference to std::__cxx is from the difference of gcc version from third-party library ( gcc 7.5.0 ubuntu 18.04) and the compiler (gcc 4.8.5 Red Hat).  I wonder if I can build third-party libraries ( static shared) on alternative gcc 4.8.5 ubuntu or not? 

Your software will have to run on our CentOS 7.6 image.  As for the difference between compiler versions, have you tried building your libraries using devtoolset (gcc 7.5.0), and running it on our standard CentOS 7.6 image (gcc 4.8.5)?

thank you for quick response. 
Currently, I am using ubuntu. I built my libraries based on Cmake gcc 7.5.0. But it failed to link to centOS 7.6 image (gcc 4.8.5) ( as shown at the beginning of this post, most errors come from the difference of gcc versions). my question is: Is there any way to build libraries based on ubuntu that can link to Nist's environment?

It's not going to work given your library is being built with a later version of gcc.  Can you build your libraries with devtoolset (e.g., https://www.softwarecollections.org/en/scls/rhscl/devtoolset-7/) on CentOS instead?

for some reason, I have to build on ubuntu, sorry about that. However, what if my lib is being built on ubuntu with older gcc version., say 4.8.5 ( or 4.9.3)? it is ok for the linking to CentOS image 7.6 ( redHat 4.8.5) ?
GitHub issue: how to link variable declared in initialize to use in createTemplate
State: closed

hello, I am interested in Nist and will submit for validation soon. Now I got in a trouble, please help.
I defined and loaded a model-param from Initialize, but I am confusing at how to call the model in createTemplate. 
Please give some advice or Am I misunderstanding somewhere? Much appreciated any help.
-- Nhat

One way to do it is to declare your model param/variable as a member variable in your implementation class - see https://github.com/usnistgov/frvt/blob/master/11/src/nullImpl/nullimplfrvt11.h for example.

Thank you, I got it. Gona close issue
GitHub issue: threshold to classify as the same person
State: closed

We are preparing a validation package for FRVT11. Log files with similarity score (range from 0.00 to 1.00) are successfully generated. May we know which threshold FRVT will use to determine a pair of template is from the same person or different. 
Or we just need document it clearly in submission email?

Much appreciated any help,

-Bill

We will run both mated (same person) and non-mated (different person) comparisons.  We expect mated scores to be high and non-mated scores to be low.  Then we will conduct ROC/DET analysis such that we sweep through a large number of score thresholds to determine what the false negative and false positive rates are at those thresholds.  So to answer your question - we won't be using any single threshold - we will analyze many thresholds.  Hope this helps.
GitHub issue: how can i set a threshold for frvt_11?
State: closed

matchTemplates(...)
just get a similarity score,i don`t know how can i set a condition for verification to judgment both of them were same people or not

The matchTemplates() function only asks that you return a similarity score, not a decision on whether comparison is of the same person or not.