ElisonSherton/github-issues huggingface dataset:
GitHub issue: Template type
State: closed

Hello @mlngan ,
I am going to participate to the challenge, but I am confusing for a problem:
About the template type, could I change the uint8_t in const std::vector<uint8_t> &verifTemplate and all related variables? The reason is my algorithm's output is in real numbers, this type cannot store the real numbers. Or I must modify my algorithm?
Thank you!!

The template type cannot be changed.  Please take a look at the provided null implementation for an example of how one might store/retrieve a vector of floats from a vector of uint8_t.

https://github.com/usnistgov/frvt/blob/master/11/src/nullImpl/nullimplfrvt11.cpp

@mlngan thank you a lot!
Sorry for bother you again. I saw and tried, but I cannot get exactly my float.
I saw that I can get maximum only 6 characters. Eg. I store 765.88989, but retrieve only 765.89.
Maybe I did a mistake?
Thank you!

> I saw that I can get maximum only 6 characters. Eg. I store 765.88989, but retrieve only 765.89.

I've got the same problem. It strongly affects algorithm accuracy. A vector of uint8_t cannot present a float-type feature
@nqkhai1706 Did you find any solutions?
@mlngan Do you have any ideas for this?

@cannguyen275 The float is being stored and retrieved correctly on my end.  Try setting the precision of the output stream when printing.  For example, try adding:
 
cout.precision(5); 
for (unsigned int i=0; i < this->featureVectorSize; i++) {
    std::cout << fixed << featureVector[i] << std::endl;
}

/* Note: example code, potentially not portable across machines. */
    std::vector<float> fv = {1.0, 2.0, 8.88, 765.88989};
    const uint8_t* bytes = reinterpret_cast<const uint8_t*>(fv.data());
    int dataSize = sizeof(float) * fv.size();
    templ.resize(dataSize);
    memcpy(templ.data(), bytes, dataSize);


From: Kei ***@***.***>
Date: Monday, May 17, 2021 at 05:50
To: usnistgov/frvt ***@***.***>
Cc: Subscribed ***@***.***>
Subject: Re: [usnistgov/frvt] Template type (#29)

I saw that I can get maximum only 6 characters. Eg. I store 765.88989, but retrieve only 765.89.

I've got the same problem. It strongly affects algorithm accuracy. A vector of uint8_t cannot present a float-type feature
@nqkhai1706<https://github.com/nqkhai1706> Did you find any solutions?
@mlngan<https://github.com/mlngan> Do you have any ideas for this?

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub<https://github.com/usnistgov/frvt/issues/29#issuecomment-842186243>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ADVZFNMIDLN4FXLLYJN6TCLTODRGJANCNFSM4MMBUH2A>.
GitHub issue: validation script doesn't link dependency libraries
State: closed

Even if I place my 3rd party dependencies (*.so) under ./lib, the compile_and_link.sh script (invoked while running run_validate_11.sh) only takes the implementation library and I get a lot of undefined references. Am I missing something? Please help.

You can use rpath=$ORIGIN, but you need to link your submission library against any 3rd party libraries that are required.  For example:

set (CMAKE_SHARED_LINKER_FLAGS "-Wl,-rpath,$ORIGIN")
add_library (frvt_11_null_001 SHARED nullimplfrvt11.cpp)
target_link_libraries(frvt_11_null_001 thirdpartylib1.so thirdpartylib2.so thirdpartylib3.so)


I have a model folder 'my_model' stored in ./lib 
But when I apply the codes above in the CMakeLists.txt in nullImpl folder

set (CMAKE_SHARED_LINKER_FLAGS "-Wl,-rpath,$ORIGIN")
add_library (frvt_11_null_001 SHARED nullimplfrvt11.cpp)
target_link_libraries(frvt_11_null_001 ${OpenCV_LIBS} ${PYTHON_LIBRARIES} my_model)

It gives an error while compiling with './scripts/build_null_impl.sh':
/usr/bin/ld: cannot find -lmy_model

Basically my embedded Python codes are calling the model directory to load into Tensorflow.


This GitHub isn't intended for helping you learn Cmake :/
Unless libmy_model.so is a library, you don't need to add it to link libraries.
GitHub issue: All faces need to be detected before submission?
State: closed

hi @mlngan  sorry for interrupt you again.
When I run the run_validate_11.sh, it seems that all the faces need to be detected before generate the submission package, the problem is: some of the face images are actually white/black blank or too small for detection or 90 angles which is also difficult for detection, I return the "FacedetectionError" but it seems unless we detect all the face images otherwise will not generate the submission package, please correct me if I am wrong. thank you 

The validation package contains these edge-case images to make sure your algorithm doesn't crash.  The validation script will only output a warning message for the images that didn't not successfully generate a template, but it will proceed to generate a submission package.  The warnings outputted are for your information in case you detect unexpected behavior.

@anguoyang Per the API: [https://pages.nist.gov/frvt/api/FRVT_ongoing_11_api_4.0.pdf], section 4.4.3, even when unable to extract features (and you return a non-successful return code), the output shall be a template that may be passed to the matchTemplates() function without error. That is, this routine must internally encode "template creation failed" and the matcher must transparently handle this.

@mlngan thank you, now it works.
Another minor question, can we put the .so file into config directory? most .so files are in ./lib , but there is also a .so file loaded with model files during initialization, could we put that .so into config directory? or we MUST put all .so into ./lib? currently we put it into config and run successfully and generated the package

@anguoyang Your primary libfrvt_11_xxx_xxx.so file needs to go into the ./lib folder.  If you have libraries that are needed by your models stored in the ./config folder, those can reside in the ./config folder.

@mlngan 
I have one additional .so file of only 4MB in the lib/ directory in addition to libfrvt_11_xxx_xxx.so. Will that be acceptable as a valid submission? Or is there a hard requirement of only one file in lib directory? 

You can have an unlimited number of files lib/


From: sam7177 ***@***.***>
Date: Wednesday, May 26, 2021 at 19:23
To: usnistgov/frvt ***@***.***>
Cc: Subscribed ***@***.***>
Subject: Re: [usnistgov/frvt] All faces need to be detected before submission? (#27)

@mlngan<https://github.com/mlngan>
I have one additional .so file of only 4MB in the lib/ directory in addition to libfrvt_11_xxx_xxx.so. Will that be acceptable as a valid submission? Or is there a hard requirement of only one file in lib directory?

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub<https://github.com/usnistgov/frvt/issues/27#issuecomment-849183804>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ADVZFNOUNCVTWX3OXW62NQ3TPV7G3ANCNFSM4LRYBO3A>.
GitHub issue: What does std::vector<float> fv mean?
State: closed

In your example code in nullImplFRVT11::createTemplate:
    
std::vector<float> fv = {1.0, 2.0, 8.88, 765.88989};
const uint8_t* bytes = reinterpret_cast<const uint8_t*>(fv.data());
int dataSize = sizeof(float) * fv.size();
templ.resize(dataSize);
memcpy(templ.data(), bytes, dataSize);

It seems that fv represent the face feature array right? however there are multi faces(const Multiface &faces), I just wonder why only one feature array here, however for the eye coordinates, it does:
    for (unsigned int i=0; i<faces.size(); i++) {
        eyeCoordinates.push_back(EyePair(true, true, i, i, i+1, i+1));
    }

from the parameter, it seems multi-face with only single feature array(std::vector<uint8_t> &templ), which I am really confusing, could you kindly help me? thank you


The code snippet with the feature vector is just an example showing one way of storing a vector of floats into a vector of uint8_t.  Regardless of the number of faces contained within the Multiface (vectors of faces), your algorithm should return a single template.  The template could be N number of feature vectors extracted out of N number of face images concatenated together, it could be a fusion of all your feature vectors into a single feature vector, we don't know or care about those details.  

Please note that for most face images, we expect only a single face to be in the photo.  In the rare event that more than one face is detected in an image, features should be extracted from the foreground face, that is, the largest face in the image.

>> The template could be N number of feature vectors extracted out of N number of face images concatenated together, it could be a fusion of all your feature vectors into a single feature vector, we don't know or care about those details.

Do faces contained within the Multiface belong to the same person @mlngan?
This information is very important to be clear because I can fuse all feature vectors into a single feature vector of the same person only.

Yes, all faces contained within the Multiface will belong to the same person.
GitHub issue: GPU is not supported?
State: closed

There is no support on GPU now, does it mean that you only accept the algorithms in CPU?
by the way, what is the CPU model, especially, does it support AVX2? thank you. @mlngan 

Correct, we only accept CPU-based algorithms now.  Yes, the processors we use support AVX2.
GitHub issue: library path
State: closed

Please explicitly set LD_LIBRARY_PATH in run_testdriver.sh so that it can be used with 3rd libraries without modification or question.

Or is it safe to use -rpath=$ORIGIN ?

Hey, I have a similar problem where I'm not able to link against 3rdparty libraries. How did you solve it? Can you help?

I set environment variable outside like: env LD_LIBRARY_PATH=lib ./scripts/run_testdriver.sh
Not sure how NIST handle this, though.

What I did to get rid of that problem is to link against 3rdpart libraries statically.
GitHub issue: Example file version.txt in folder doc 
State: closed

Is there an example of filling in a version.txt file in the doc folder ?

I don't have an example.  Version.txt should contain vendor-specific information on the specific algorithm that is being sent for testing.

Thanks for the help.

@kotbegemot have you solved it? it is Okay that just put, for example 1.0.0 in the first line of Version.txt? @mlngan 

yes - the information is supposed to keep track of the version of your algorithm submitted.
GitHub issue: A limitation of the library on a thread of execution
State: closed

We use mxnet to implement machine learning. 
We have made sure that mxnet does not generate a large number of threads for calculations.

However, we are showing a warning from the verification script :

```bash
[WARNING] We've detected that your software may be threading or using other multiprocessing techniques during template creation.  The number of threads detected was 3 and it should be 2.  Per the API document, implementations must run single-threaded.  In the test environment, there is no advantage to threading, because NIST will distribute workload across multiple blades and multiple processes.  We highly recommend that you fix this issue prior to submission.

```
How critical is this warning ?


If the additional thread from mxnet isn't doing any work, it should be fine.  We will assess the threading situation on a case-by-case basis.

Thanks for the help.
GitHub issue: Gcc version 
State: closed

Can I update the compiler version to gcc 5 ?
can I switch validate11 builds to C++ 14 ?

If you need to build your libraries with gcc 5, have you tried building your libraries using devtoolset and running it on our standard CentOS 7.6 image?

I checked on CentOS 7 there is DevTools-7. https://www.softwarecollections.org/en/scls/rhscl/devtoolset-7/
I found a workaround we learned how to build gcc 4.9.2 .
can the project be compiled by gcc 4.9.2 and submitted to the competition ?

If your submission is built with devtoolset and runs on the CentOS 7.6 image on our validation package, it should be fine.

Thanks for the help.
GitHub issue: Do I need to prove my status while submitting?
State: closed

We are going to participate in FRVT. Application form says that developers are eligible to participate. Should we somehow prove that we are developers and algorithm is ours?
Thank you.

@deidonum I have the same question, have you get feedback?

You need to complete the Application Form and send it to us. Please email us when you do. You can send your algorithm at any time. We will not evaluate it unless we receive the completed Application Form.
GitHub issue: ISO image link is down.
State: closed

Hello,

Could you please update the Centos' iso link please? The current one is not working.

Cheers,
ML

Hi ML,

Can you tell us where is the broken link?  If it is in the API document, which API document? or if is in the website, which page were you visited?  Thank you.

NIST

ML - the ISO link referenced from the validation package README file works for me - http://nigos.nist.gov:8080/evaluations/CentOS-7-x86_64-Everything-1810.iso.  Can you please try again.

The link in the API document works for me too. This Centos ISO is not
customized by NIST, and the exact same version (7.6.1810, x86_64) can be
downloaded from the CentOS archives:
http://archive.kernel.org/centos-vault/7.6.1810/isos/x86_64/

On Thu, Dec 5, 2019 at 2:26 PM mlngan <notifications@github.com> wrote:

> ML - the ISO link referenced from the validation package README file works
> for me -
> http://nigos.nist.gov:8080/evaluations/CentOS-7-x86_64-Everything-1810.iso.
> Can you please try again.
>
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/usnistgov/frvt/issues/18?email_source=notifications&email_token=AEAG2UJOR436QOOYIWLG443QXE2UDA5CNFSM4JV2UGLKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEGBPE4A#issuecomment-562229872>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AEAG2ULWCLZIMUS3OPLRNRDQXE2UDANCNFSM4JV2UGLA>
> .
>


-- 

-- Felipe


The link works for me if I remove port number:
http://nigos.nist.gov/evaluations/CentOS-7-x86_64-Everything-1810.iso

Thanks everyone!
GitHub issue: Compiler version
State: closed

Can I use g++ version greater than 4.8.5? Some libraries do not work with this version of compiler. libtorch works well with g++ 4.9.2 from devtoolset-3-gcc-c++ but breaks when using 4.8.5. https://github.com/pytorch/pytorch/issues/14981

Have you tried building your libraries using devtoolset (gcc 4.9.2), and running it on our standard CentOS 7.6 image (gcc 4.8.5)?

Thanks for the advice. I was able to successfully build the library on g++ 4.9.2 and use it with g++ 4.8.5. I think the issue can be closed.
GitHub issue: Time assessment for template creation
State: closed

Is it possible to have a brief description about the procedure used to assess the time required to create a template?
More specifically, I'm interested in understanding the batch size (namely, how many images are tested before calling `initialize` function again).

Thank you.

For timing, we call 1) initialize() 2) createTemplate() with single image inputs multiple times.  Initialize isn't called again after the first call to createTemplate().

Hi @mlngan 

If a few of the samples take more than 1500 ms for template creation, would it mean the template computation would be terminated abruptly or does it mean template would be computed but not considered for the scoring?

Consider if 10 samples are taking 1510 ms for template creation due to some issue, will those 10 samples be considered in scoring process or not?

Thanks,
Vinayak

During our timing test, we look at the median timing value across the total number of template creation invocations.  If the median value is more than 1500ms, we will notify you to fix the issue.  If it is less than or equal to 1500ms, we will proceed with evaluation.

Thank you so much! Got it.
GitHub issue: Link of Centos 7.6 is dead
State: closed

The link in the README.txt seems dead.
(http://nigos.nist.gov:8080/evaluations/CentOS-7-x86_64-Everything-1810.iso)

I just tried it this morning, and it works.  Please try again.
GitHub issue: Agreement to participate
State: closed

Is it enough to sign [this document](https://www.nist.gov/sites/default/files/documents/2019/04/22/frvt_1n_participation_agreement.pdf) for participation in 1v1 and 1vN or is it needed to sign [this](https://www.nist.gov/sites/default/files/documents/2016/12/15/frvt_participation_agreement.pdf) document too?

You need to sign separate participation agreements for 1:1 versus 1:N.

Currently we are going to participate in **1:N** and already signed and sent by post the signed document https://www.nist.gov/sites/default/files/documents/2019/04/22/frvt_1n_participation_agreement.pdf (the one that is mentioned on 1:N page). 

As we need to pass some 1:1 thresholds, do we need to **additionally** sign and send the document mentioned on **1:1** page (we don't want participate in 1:1, at least for now) https://www.nist.gov/sites/default/files/documents/2016/12/15/frvt_participation_agreement.pdf ?

As published on our 1:N website:
In order to participate in FRVT 1:N, participants must submit at least one algorithm to the FRVT 1:1 track, and its accuracy must exceed at least one of the following:

VisaMC dataset: FNMR of 0.025 or less at FMR=0.0001 OR
Mugshot dataset: FNMR of 0.025 or less at FMR=0.00001

Once an organization submits a 1:1 algorithm that exceeds the minimum accuracy requirements, that organization may submit algorithms to the 1:N track thereafter.

hi @mlngan ,
as your mentioned, in order to participate FRVT 1:N, the FRVT 1:1 accuracy must pass some requirements. But I find that NEC performs very well on the FRVT 1:N leaderboard, but they are empty from FRVT 1:1 leaderboard. 
How was it? thank you.

Hi - we removed the 1:1 submission requirement in order to participate in 1:N some time ago, so developers can submit directly to the 1:N track if they choose.
GitHub issue: Install required packages for runtime
State: closed

Is it possible to install dependent packages for runtime and list these packages in ./config directory?
For example, I am implementing with OpenCV and OpenCV requires these packages installed for runtime:
- libjpeg-turbo-devel
- libtiff-devel 
- libdc1394-devel
- eigen3-devel
and etc.

For any software dependencies your algorithm requires, you need to provide them (the libraries themselves) in your submission under the ./lib folder.

hi @mexeniz, @mlngan 
how did you handle the libraries needed by third-party libraries?
for example: 
set (CMAKE_SHARED_LINKER_FLAGS "-Wl,-rpath,$ORIGIN")
add_library (frvt_11_null_001 SHARED nullimplfrvt11.cpp)
target_link_libraries(frvt_11_null_001 libdc1394.so.22 thirdpartylib2.so ...)
We can submit libdc1394.so.22 in ./lib folder but  libdc1394.so.22 also call to some other libs like: libraw1394.so.0.1 ,... .
How did you handle these cases? Thank you in advance
GitHub issue: PYTHONPATH, PYTHONHOME
State: closed

Hello. I want to participate in FRVT(1v1 and 1vN). My SDK uses Python interpreter and has Python code. In order to use Python my SDK must have runtime access to Python dependencies: a lot of *.py, *.so and other files. And before using Python it is needed to initialize PYTHONPATH and other paths.

My question: is it allowed to place several *.py and *.so files into config directory?

You may place those files in ./config or perhaps a better place for them would be ./lib.  Please keep in mind ./config will be a READ-ONLY directory.
GitHub issue: Fork return fail
State: closed

Hi, 

For some reason our current compilers fail to join at forked processes at validate11.cpp with return statement. It appears that using exit instead works. Any idea if this is an indication of some problem on our side or can this change be done safely?

Ie:
```
switch(fork()) {
		case 0: /* Child */
			if (action == Action::CreateTemplate)
				return createTemplate(
						implPtr,
						inputFile,
						outputDir + "/" + outputFileStem + ".log." + to_string(i),
						templatesDir,
						role);
			else if (action == Action::Match)
				return match(
						implPtr,
						inputFile,
						templatesDir,
                                                outputDir + "/" + outputFileStem + ".log." + to_string(i));
```
to 

```
switch(fork()) {
		case 0: /* Child */
			if (action == Action::CreateTemplate)
				exit( createTemplate(
						implPtr,
						inputFile,
						outputDir + "/" + outputFileStem + ".log." + to_string(i),
						templatesDir,
						role));
			else if (action == Action::Match)
				exit( match(
						implPtr,
						inputFile,
						templatesDir,
						outputDir + "/" + outputFileStem + ".log." + to_string(i)));
```

I can confirm no such issue here.

My guess is that exit is avoiding the issue of running some destructors or killing some threads, which would be handled (and possibly fail) with the return method.

@svarjo how did you solve this problem? I got into the same as you are. child process does not return anything ( no bug) and that is why the whole process is suspended.

@quocnhat - If I recall right it was some release bug on our side of code. Maybe it is a good idea to verify memory usage with some tool such as AddressSanitizer or Valgrind.
GitHub issue: Unable to download Centos image
State: closed

Hi,

I was unable to download the Centos image through the provided link in the README file: http://nigos.nist.gov:8080/evaluations/CentOS-7-x86_64-Everything-1511.iso

Best reagrds,
Vitor Pedro
GitHub issue: Pb with quality when no face is detected
State: closed

The code below write template stats to log
```
/* Write template stats to log */
        logStream << id << " "
                << imagePath << " "
                << templ.size() << " "
                << static_cast<std::underlying_type<ReturnCode>::type>(ret.code) << " "
                << (eyes.size() > 0 ? eyes[0].isLeftAssigned : false) << " "
                << (eyes.size() > 0 ? eyes[0].isRightAssigned : false) << " "
                << (eyes.size() > 0 ? eyes[0].xleft : 0) << " "
                << (eyes.size() > 0 ? eyes[0].yleft : 0) << " "
                << (eyes.size() > 0 ? eyes[0].xright : 0) << " "
                << (eyes.size() > 0 ? eyes[0].yright : 0) << " "
                << quality[0]
                << endl;
```
But when no face is detected, size of vector of quality is 0, so quality[0] will crash

A check has been added for the size of the quality vector.
GitHub issue: 1:n validation package not completely updated
State: closed

The frvt1N.h has been updated for the function finalizeEnrollment, but validation(/validation/src/testdriver/validate1N.cpp) wasn't updated, could you help to check and update the validation for 1:n phase3? Thanks.

Validation package has been updated to reflect changes to finalizeEnrollment() for phase 3.
GitHub issue: compilation warning
State: closed

In frvt11.h, the lack of a default label in the method

`inline std::ostream& operator<<(std::ostream &s, const ReturnCode &rc)`

causes a compiler warning when it is called:

`c++ -std=c++11 -mavx -Wall -O2 -c trivial.cc -o libtrivial.so`
`c++ -std=c++11 -mavx -Wall -O2 client.cc -L. -ltrivial -o client`
`In file included from client.cc:2:0:`
`frvt11.h: In function ‘std::ostream& FRVT::operator<<(std::ostream&, const FRVT::ReturnCode&)’:`
`frvt11.h:163:1: warning: control reaches end of non-void function [-Wreturn-type]`



Added default statement to ReturnCode output stream operator.