ElisonSherton/github-issues huggingface dataset:
GitHub issue: Possibility to relax Ubuntu 20.04 minor version requirement
State: open

Hi,

I am wondering whether it is necessary to specifically check for Ubuntu version 20.04.3 LTS.
The currently supported version is Ubuntu 20.04.6 and contains lots of important security updates.
Since the minor version changes should not have a relevant impact on the included packages, I would advocate to allow all Ubuntu 20.04.x versions.

I will create a PR and hope that the PR can be accepted.

Kind regards
GitHub issue: incorrect path to some images for age-estimation
State: closed

Hi, the commit 18038d652b950f919f53957b9ddda8d34fd7d04b add folder "videos" to all image paths in the singleMedia.txt file, however some images are in the `common/images/face/` folder

eg:

`../common/images/face/videos/small.ppm` should be  `../common/images/face/small.ppm`
GitHub issue: Unexpected path of `video1` for `run_validate_ae.sh`
State: closed

When running `./run_validate_ae.sh`, the execution fails with:
![image](https://github.com/usnistgov/frvt/assets/52961799/ba30b30c-6a18-4efd-ad18-88b48591c287)

This can be fixed by moving the `video1` folder down by one directory, however, this should not be expected of the user.

We've updated the file with the correct path.  Thank you.
GitHub issue: Disparity of median enrollment time when running on single core.
State: closed

The NIST document specifies that the median enrollment time must be under 1.5 seconds on a single core, tested on an Intel(R) Xeon(R) Gold 6140 CPU @ 2.30GHz. In the run_testdriver.sh script, the num_forks argument is used to run validate11.cpp.

When I run the script on a single core using the command taskset --cpu-list 1 run_validate11.sh, the enrollment time for multiple processes (i.e., num_forks = 4) increases fourfold compared to a single process. However, there is no increase in time for multiple processes when I run the script without the taskset command.

How does NIST internally check the median enrollment time? Do they use the taskset command to limit the number of CPUs to one?

During our timing test, we only run with a single process (and limit the number of CPUs to one).  We run and time the template generation call over many iterations, then take the median of those timing values.
GitHub issue: Suggest to split and zip "common/images/face" to release files
State: open

now git clone the frvt repo would create 10G+ .git files (files in frvt repo not counting yet), while the scripts and source code files only a few KBs. 

it is not a good idea to git clone the whole frvt repo while you are mixing the data files (no version control/history required) and source files in git.

`git clone --depth 1` may lower the size of files

thank you for the approach. but for my trying, still have 7.7G. the images and video size is 5.7G, cannot avoid that huge .git folder. 

I think basically caused by 4.7G videos uploaded recently by this commit 7ca40ad12d548232e88aecb167934bac1e659f29.
GitHub issue: Doubt regarding NIST Public Key
State: closed

Hi!

I am trying to encrypt my final submission file using the outlines [mentioned here](https://www.nist.gov/system/files/documents/2021/10/25/nist_encryption.pdf).

As mentioned in section 2.3, I try to import the NIST public key for FRTE [provided here](https://www.nist.gov/system/files/documents/2021/10/25/nist_encryption.pdf) in the second row.

However the key that I get is slightly different. The fingerprint obtained after importing the key is the same as the one mentioned in submission document but the key is different as can be seen below.

![key_mismatch](https://github.com/usnistgov/frvt/assets/45599762/48e32547-5bc7-4be4-8e41-0e0f980431d4)

Is this acceptable for the submission or am I missing something when importing the key?

Thanks,
Vinayak.



If you are saying the string 856B9B28 does not match the string 5A18F9... then that is expected

If you are saying that something else doesn't match. then email us again.
________________________________
From: Vinayak Nayak ***@***.***>
Sent: Friday, May 24, 2024 3:23 AM
To: usnistgov/frvt ***@***.***>
Cc: Subscribed ***@***.***>
Subject: [usnistgov/frvt] Doubt regarding NIST Public Key (Issue #134)


Hi!

I am trying to encrypt my final submission file using the outlines mentioned here<https://www.nist.gov/system/files/documents/2021/10/25/nist_encryption.pdf>.

As mentioned in section 2.3, I try to import the NIST public key for FRTE provided here<https://www.nist.gov/system/files/documents/2021/10/25/nist_encryption.pdf> in the second row.

However the key that I get is slightly different. The fingerprint obtained after importing the key is the same as the one mentioned in submission document but the key is different as can be seen below.

key_mismatch.png (view on web)<https://github.com/usnistgov/frvt/assets/45599762/48e32547-5bc7-4be4-8e41-0e0f980431d4>

Is this acceptable for the submission or am I missing something when importing the key?

Thanks,
Vinayak.

—
Reply to this email directly, view it on GitHub<https://github.com/usnistgov/frvt/issues/134>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ADVZFNO35GUQN6DQLHTUSJ3ZD3TGVAVCNFSM6AAAAABIHA4MCSVHI2DSMVQWIX3LMV43ASLTON2WKOZSGMYTINRSGI2DKMI>.
You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>


Yeah that's what I meant. Thanks for the confirmation!
GitHub issue: Failure-to-enroll/-to-extract
State: open

Hi,

I have a question regarding the evaluation and reporting of failure-to-enroll and failure-to-extract. Your report on page 40 mentions that failure-to-extract (the inability to produce a template for search) is not reported. However, the table header in the results section reads "Failure to extract features." Could you clarify if this actually refers to failure-to-enroll, which is the inability to produce a template for enrollment? Additionally, are the same evaluation metrics used for both Face and Iris in IREX10, including the failures mentioned above?

Page 40 in which report (?).   In our evaluations a failure to convert an image to a template will elevate 1:1 FNMR or 1:N FNIR if the template is used in a mated comparison.  It will not cause increase 1:1 FMR or 1:N FPIR.   The effect is the same as if the template had been produced but it always produces a low similarity score.







From: Erik ***@***.***>
Date: Monday, May 13, 2024 at 11:34 AM
To: usnistgov/frvt ***@***.***>
Cc: Subscribed ***@***.***>
Subject: [usnistgov/frvt] Failure-to-enroll/-to-extract (Issue #133)

Hi,

I have a question regarding the evaluation and reporting of failure-to-enroll and failure-to-extract. Your report on page 40 mentions that failure-to-extract (the inability to produce a template for search) is not reported. However, the table header in the results section reads "Failure to extract features." Could you clarify if this actually refers to failure-to-enroll, which is the inability to produce a template for enrollment? Additionally, are the same evaluation metrics used for both Face and Iris in IREX10, including the failures mentioned above?

—
Reply to this email directly, view it on GitHub<https://github.com/usnistgov/frvt/issues/133>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ADVZFNITMKXX2P4N3YTSIY3ZCDMRZAVCNFSM6AAAAABHUOQJ5GVHI2DSMVQWIX3LMV43ASLTON2WKOZSGI4TGMJVHE4DSMY>.
You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>


I came across this issue before. The wording is a bit misleading, and it suggests it isn't counted.
Think of it logically: Of course it is counted in the results. Otherwise, it is in your interest to fail to extract any marginally difficult images.

It is counted and will be considered the worst score, so it is in your interest to actually extract it usually. You get given enough time to try various methods of extraction.

@paddygr : I'm referring to the latest FRTE 1:N report, specifically section 3.5, which states, "We do not report FTX...". Given this, I assume the numbers reported in Table 15 under the column "failure to extract features" are only FTE – the proportion of templates that could not be generated during enrollment. My follow-up question concerns the IREX10 submissions: Does the FTE Rate reported on the table on the website also refer to the proportion of template generation during enrollment?
GitHub issue: Linking external so libraries using CMAKE_INSTALL_RPATH.
State: closed

I want to link the "libfrvt_1N_null_000.so" (present in "./lib" folder) with the opencv and onnxruntime libraries which are present in "./lib/opencv" and "./lib/onnxruntime" folder respectively. How can I link the main .so file with the opencv and onnx dependencies such that relative path is added using CMake. I tried using  CMAKE_INSTALL_RPATH but I am still unable to link it as relative path.

Thankyou

This thread may be helpful to you - https://github.com/usnistgov/frvt/issues/128

I followed the thread but still unable to link the libfrvt_1N_null_000.so file with the dependencies as relative path. Following is the folder structure and the cmake file:
[CMakeLists.txt](https://github.com/usnistgov/frvt/files/14534503/CMakeLists.txt)
![nist1](https://github.com/usnistgov/frvt/assets/85778019/bae289f4-0fcb-44d8-84da-b74a8087efcb)

The onnx and opencv ".so" files are in onnxruntime/build/Linux/Release/ and opencv/build/lib/ folders
What changes should I make in CMakeLists.txt in order to add relative path of onnx and opencv so files?
In the thread you mentioned, he moves all the opencv .so files in lib folder and sets the as the LDD_Library_path is set to frvt/1N/lib. I also did the same but couldn't achieve relative linking.



I use onnxruntime and opencv. This is my `CMakeLists.txt`. You can refer to it.
```
project(nullImpl)
cmake_minimum_required(VERSION 2.8)

# Find OpenCV package
# set(CMAKE_MODULE_PATH ${CMAKE_MODULE_PATH} ${CMAKE_CURRENT_SOURCE_DIR}/../../lib/opencv-4.9.0/cmake)
set(OpenCV_DIR ${CMAKE_CURRENT_SOURCE_DIR}/../../lib/opencv-4.9.0/lib/cmake/opencv4)
find_package(OpenCV REQUIRED)

set (CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -std=c++17")

# Link onnxruntime
set(ONNX_LIB_DIR ${CMAKE_CURRENT_SOURCE_DIR}/../../lib/onnxruntime-linux-x64-1.17.1/lib)

set(SOURCES nullimplfrvt11.cpp fd.cpp)
add_executable(nullimplfrvt11 ${SOURCES})
target_link_libraries(nullimplfrvt11 ${ONNX_LIB_DIR}/libonnxruntime.so 
                        			 ${ONNX_LIB_DIR}/libonnxruntime.so.1.17.1)
target_link_libraries(nullimplfrvt11 ${OpenCV_LIBS})

# Include directories
include_directories(${OpenCV_INCLUDE_DIRS})
include_directories(${CMAKE_CURRENT_SOURCE_DIR}/../include 
                    ${CMAKE_CURRENT_SOURCE_DIR}/../../../common/src/include
                    ${CMAKE_CURRENT_SOURCE_DIR}/../../lib/onnxruntime-linux-x64-1.17.1/include
                    ${CMAKE_CURRENT_SOURCE_DIR}/../../lib/opencv-4.9.0/include)

# Configure built shared libraries in top-level lib directory
set (CMAKE_LIBRARY_OUTPUT_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}/../../lib)

# Build the shared libraries
add_library (frvt_11_VVTK_001 SHARED nullimplfrvt11.cpp fd.cpp)

# Link Torch to the shared library
target_link_libraries(frvt_11_VVTK_001 ${ONNX_LIB_DIR}/libonnxruntime.so 
                        			   ${ONNX_LIB_DIR}/libonnxruntime.so.1.17.1)
target_link_libraries(frvt_11_VVTK_001 ${OpenCV_LIBS})
```
GitHub issue: FRTE Latest Report Query
State: closed

In the [latest report here](https://pages.nist.gov/frvt/reports/11/frvt_11_report.pdf), under section 3.2, the test design principle for different types of comparison is provided.

In all the tracks (namely visa, mugshot, visa-border, visa-border non-frontal yaw, kiosk-border, border-border, wild, child-exploitation), the following two statements are present:

> The number of images used to make a template is one.
> The number of templates used to make each comparison score is two corresponding to simple one-to-one verification

In the [API document](https://pages.nist.gov/frvt/api/FRVT_ongoing_11_api.pdf) on line 82, we can see the following table

![image](https://github.com/usnistgov/frvt/assets/45599762/cac32890-5639-4b75-92a4-f15485c69542)

My question is then when will multiple images (i.e. K > 1) be sent to `createFaceTemplate` API for making an enrolment or verification template?

Thanks,
Vinayak.

Hi @mlngan 

If you can provide some clarification here, it would be really great!

Thanks,
Vinayak.

K > 1 is not exercised currently.  It will be in the future.


From: Vinayak Nayak ***@***.***>
Date: Thursday, March 7, 2024 at 7:34 AM
To: usnistgov/frvt ***@***.***>
Cc: Subscribed ***@***.***>
Subject: [usnistgov/frvt] FRTE Latest Report Query (Issue #129)

In the latest report here<https://pages.nist.gov/frvt/reports/11/frvt_11_report.pdf>, under section 3.2, the test design principle for different types of comparison is provided.

In all the tracks (namely visa, mugshot, visa-border, visa-border non-frontal yaw, kiosk-border, border-border, wild, child-exploitation), the following two statements are present:

The number of images used to make a template is one.
The number of templates used to make each comparison score is two corresponding to simple one-to-one verification

In the API document<https://pages.nist.gov/frvt/api/FRVT_ongoing_11_api.pdf> on line 82, we can see the following table

image.png (view on web)<https://github.com/usnistgov/frvt/assets/45599762/cac32890-5639-4b75-92a4-f15485c69542>

My question is then when will multiple images (i.e. K > 1) be sent to createFaceTemplate API for making an enrolment or verification template?

Thanks,
Vinayak.

—
Reply to this email directly, view it on GitHub<https://github.com/usnistgov/frvt/issues/129>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ADVZFNOEFLOKP6YEM3DK3MLYXBNBFAVCNFSM6AAAAABEK5CTMKVHI2DSMVQWIX3LMV43ASLTON2WKOZSGE3TGNZWGA3TCNQ>.
You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>
GitHub issue: free(): invalid pointer after trying to link torch library
State: open

I encounter the same issue as #49 . I simply add the following line in `frvt/11/src/nullImpl/CMakeLists.txt`:
```
target_link_libraries(frvt_11_null_001 ${TORCH_LIBRARIES})
```

Then, running `./scripts/build_null_impl.sh` and `./run_validate_11.sh` gives
```
Checking for hard-coded config directory free(): invalid pointer
PID 33486 exited due to signal 6
free(): invalid pointer
```
`./run_validate_11.sh` run successfully without that line.
How can I link libtorch correctly in the `CMakeLists.txt`? Thanks for your help!
GitHub issue: ONNX Runtime Multithreading Issue Preventing NIST FRVT 11 Submission
State: open

## Reporting a Multithreading Issue in ONNX Runtime to NIST FRVT GitHub Repository
**Body:**

I am encountering an issue with ONNX Runtime that prevents my submission to the NIST FRVT 11 evaluation. 

**Describe the problem:**

* I am using ONNX Runtime for inference in my submission package.
* I have attempted to restrict the number of threads used by ONNX Runtime to 1 using the provided GUID.
* However, when my submission package is executed, the NIST FRVT script detects 17 threads instead of the expected 2.
* This discrepancy becomes apparent when using the `top -H` command, which reveals 17 threads are created, with only 1 in a running state and the remaining 16 in a sleeping state.
* Due to this multithreading behavior, my package crashes when executed by NIST FRVT on multiple cores, violating their single-threaded requirement.

**Here is screen shot of htop of same package**
<img width="937" alt="Screenshot 2024-02-28 at 7 03 14 PM" src="https://github.com/usnistgov/frvt/assets/97581451/945394ba-3486-4a44-9b44-924692796293">

* Only one thread is running but why there are so many threads launched

**Possible impact:**

* This issue prevents my package from being successfully evaluated in the NIST FRVT 11 program.


* I kindly request the NIST FRVT developers to investigate this multithreading issue in ONNX Runtime and provide any potential solutions or workarounds.
* Additionally, I would appreciate guidance on overcoming this limitation and ensuring successful submission compliance with the single-threaded requirement.

**Additional information:**

* **Version of ONNX Runtime==1.10.0**

**Closing:**

Thank you for your time and consideration. I look forward to your response and assistance in resolving this issue.

Did you close the docker and re-open one?

there is no usage of docker here. It ran directly on host machine as recommended.
GitHub issue: Error in compiling with libtorch
State: closed

I download libtorch by `wget https://download.pytorch.org/libtorch/nightly/cpu/libtorch-win-shared-with-deps-latest.zip`.

I add the following code inside `nullimplfrvt11.cpp`.
```C++
#include <torch/script.h>

ReturnStatus
NullImplFRVT11::initialize(const std::string &configDir)
{
    torch::jit::script::Module module;
    try {
	    std::string path = configDir + "script_model.pt";
	    module = torch::jit::load(path);
    }
    catch (const c10::Error& e) {
	    std::cerr << "error loading the model" << e.msg() << "\n";
	    return ReturnStatus(ReturnCode::ConfigError);
    }
    std::cout << "ok\n";
    return ReturnStatus(ReturnCode::Success);
}
```
Running `./script/build_null_impl.sh` was fine and generated `libfrvt_11_<>_001.so`.
But when I run `./run_validate_11.sh`, it has the following error:
```
/usr/bin/ld: /train/frvt/11/lib/libfrvt_11_VVTK_001.so: undefined reference to `c10::ListType::get(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, c10::Type::SingletonOrSharedTypePtr<c10::Type>)'
/usr/bin/ld: /train/frvt/11/lib/libfrvt_11_VVTK_001.so: undefined reference to `torch::jit::load(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, c10::optional<c10::Device>, bool)'
/usr/bin/ld: /train/frvt/11/lib/libfrvt_11_VVTK_001.so: undefined reference to `c10::StringType::get()'
/usr/bin/ld: /train/frvt/11/lib/libfrvt_11_VVTK_001.so: undefined reference to `c10::AnyType::get()'
/usr/bin/ld: /train/frvt/11/lib/libfrvt_11_VVTK_001.so: undefined reference to `c10::UndefinedTensorImpl::_singleton'
```

I have followed [pytorch/pytorch#13541](https://github.com/pytorch/pytorch/issues/13541) and [pytorch/pytorch#17492](https://github.com/pytorch/pytorch/issues/17492). But still no luck. Any suggestions or something I've missed? Thanks for your help.

revise the content of `./src/nullImpl/CMakeLists.txt` solved the issue.
```
project(nullImpl)
cmake_minimum_required(VERSION 2.8)

find_package(Torch REQUIRED)
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} ${TORCH_CXX_FLAGS}")

add_executable(nullImpl nullimplfrvt11.cpp)
target_link_libraries(nullImpl ${TORCH_LIBRARIES})
set_property(TARGET nullImpl PROPERTY CXX_STANDARD 17)

set (CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -std=c++17")
include_directories (${CMAKE_CURRENT_SOURCE_DIR}/../include ${CMAKE_CURRENT_SOURCE_DIR}/../../../common/src/include ${CMAKE_CURRENT_SOURCE_DIR}/../../lib/libtorch/include)

# Configure built shared libraries in top-level lib directory
set (CMAKE_LIBRARY_OUTPUT_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}/../../lib)

# Build the shared libraries
add_library (frvt_11_VVTK_001 SHARED nullimplfrvt11.cpp)
target_link_libraries(frvt_11_VVTK_001 ${TORCH_LIBRARIES})
```
GitHub issue: Submission of Dynamically linked library(.so file) to NIST.
State: closed

Hi I have used onnxruntime and opencv libraries in my face recognition implementation. NIST asks vendors to submit a single .so file (libfrvt_<track>_<provider>_<sequence>.so) (core implementation) as a part of tar package. The only library that will be explicitly linked to test driver is the core library. Currently I am working in docker environment that has opencv and onnxruntime dependencies and generating the .so file. However, I am unable to run the testdriver script outside the docker environment as no dependencies are there. 
Can I provide supplemental .so libraries (containing the dependencies) along with the core .so file? If yes then how?

Yes, you may include supplemental .so libraries in the same ./lib folder where your core library is located.
GitHub issue: Conversion of uint8_t to float for template
State: closed

Hi! I have a doubt regarding the conversion of unint8_t to float for the template vector. In the validate1N.cpp file templ(template) variable datatype is declared as vector<uint8_t>. The datatype of embeddings from the recognizer model is of data type float. How can I obtain a conversion of uint8 to float. Also, if I use reinterpret cast for conversion, the values I get are incorrect. Why havent NIST utilized data type of templ as float or double instead of uint8_t?



Many developers use floating point internally, but we use a generic byte stream to allow any type, including a constructed type.  One solution is to use casting, for example.
 
// Cast float to uint8_t during template creation
const uint8_t* bytes = reinterpret_cast<const uint8_t*>(fv.data());
      
// Cast uint8_t back to float during match
float *featureVector = (float *)enrollTemplate.data();
 
Can you elaborate on what you mean by the values are incorrect when you use reinterpret_cast?

Thanks! Above solution worked for me!
GitHub issue: CMake version
State: open

my Lib c++ is used CMake ver 3.27. and when submit lib, your system support it ?
GitHub issue: Error when runnung the run_validate_11.sh.  [ERROR] Match validation failed
State: closed

Hi there,

I'm trying to pack up the submit code. I can run the null implementation and pack up the null testing code successfully. 

But if I try to pack up my own submitted code, there will have some error like follows: 

Could anyone figure out the problem?

```
[100%] Built target validate11
[SUCCESS] Built executable in /home/pj/frvt_2023_adaface_submit/11/bin.
------------------------------
 Running 1:1 validation
------------------------------
Checking for hard-coded config directory [SUCCESS]
Creating Enrollment Templates (Single Process) [SUCCESS]
Creating Enrollment Templates on Multiple Images per Subject (Single Process) [SUCCESS]
Creating Enrollment Templates (Multiple Processes) PID 4474 exited due to signal 9
PID 4473 exited due to signal 9
[SUCCESS]
Creating Verification Templates (Multiple Processes) PID 4491 exited due to signal 9
PID 4493 exited due to signal 9
[SUCCESS]
Matching Templates (Multiple Processes) [ERROR] Failed to open stream for validation/templates/332.template.
[ERROR] Unable to retrieve template from file : validation/templates/332.template
[ERROR] Failed to open stream for validation/templates/657.template.
[ERROR] Unable to retrieve template from file : validation/templates/657.template
PID 4513 exited due to signal 15
[ERROR] Failed to open stream for validation/templates/985.template.
[ERROR] Unable to retrieve template from file : validation/templates/985.template
PID 4514 exited due to signal 15
PID 4515 exited due to signal 15
[ERROR] Match validation failed
```


[Update] Problem solved!

Because I was using virtual machine to run the FRVT validationand the error is due to the small number of system processor and not enough of space of the memory capacity.

If increace the number of system processor and memory capacity, the problem can be solved and successfully go through all the validation process.
GitHub issue: why is Finalization function's time limited 40000 sec ?
State: open

enroll function in validate1N.cpp call createTemplate from list ppm in input file then write into edb file and manifest. 
My createTemplate function, I created Feature Vector.
I used initializeIdentification func to read edb and manifest, init search algorithms when Search Initialization
I think finalization is used to re-write edb and manifest.




Time is limited because a developers might choose to do an expensive computation instead of a copy.


From: shritor ***@***.***>
Date: Wednesday, November 1, 2023 at 6:17 AM
To: usnistgov/frvt ***@***.***>
Cc: Subscribed ***@***.***>
Subject: [usnistgov/frvt] why is Finalization function's time limited 40000 sec ? (Issue #114)

enroll function in validate1N.cpp call createTemplate from list ppm in input file then write into edb file and manifest.
my createTemplate function, I created Feature Vector.
and I used initializeIdentification func to read edb and manifest, init search algorithms when Search Initialization
I think finalization is used to re-write edb and manifest.

—
Reply to this email directly, view it on GitHub<https://github.com/usnistgov/frvt/issues/114>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ADVZFNLPKKDKPW63BJYY4ELYCIOTXAVCNFSM6AAAAAA6ZB3KI2VHI2DSMVQWIX3LMV43ASLTON2WKOZRHE3TEMBYGQ4DQOI>.
You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>


@paddygr 
![Screenshot from 2024-06-05 11-27-59](https://github.com/usnistgov/frvt/assets/149577702/e75a8fac-96e7-4e37-941c-26c2c73ec71a)
Can you tell me why there is no result of 3M and 12M vector ? alttekglobal_000

Accuracy is very poor; it’s not a good use of resources to run at large N.

From: shritor ***@***.***>
Date: Sunday, June 9, 2024 at 9:48 PM
To: usnistgov/frvt ***@***.***>
Cc: Grother, Patrick J. (Fed) ***@***.***>, Mention ***@***.***>
Subject: Re: [usnistgov/frvt] why is Finalization function's time limited 40000 sec ? (Issue #114)

@paddygr<https://github.com/paddygr>
Screenshot.from.2024-06-05.11-27-59.png (view on web)<https://github.com/usnistgov/frvt/assets/149577702/e75a8fac-96e7-4e37-941c-26c2c73ec71a>
Can you tell me why there is no result of 3M and 12M vector ? alttekglobal_000

—
Reply to this email directly, view it on GitHub<https://github.com/usnistgov/frvt/issues/114#issuecomment-2157033832>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ADVZFNN2KAPINBRMOVJKLEDZGUAVBAVCNFSM6AAAAABJBMK5BKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNJXGAZTGOBTGI>.
You are receiving this because you were mentioned.Message ID: ***@***.***>
GitHub issue: [FRVT 1:1] Can I edit the header file? 
State: closed

nullimplfrvt11.h
 I fix it from uint8_t to float.
![Screenshot from 2023-10-23 11-17-58](https://github.com/usnistgov/frvt/assets/55545444/463cad6c-95ac-4e06-a5f6-96f09b1065db)


You cannot modify any of the source code that we have published.  Please consider the example provided in https://github.com/usnistgov/frvt/blob/master/11/src/nullImpl/nullimplfrvt11.cpp on how to convert between uint8_t and float.

Thanks for your answer, in file cpp, matching is random. Can I modify it.
![Screenshot from 2023-10-24 08-15-49](https://github.com/usnistgov/frvt/assets/55545444/cc8014f3-000b-4354-b6f3-d7199a5f53ae)
What are editable files?, util.h, util.cpp,validate11.cpp ?

You implement whatever match you want, in your compiled library.  You must not change the API.  You cannot edit those files.

From: Hoàng Bá Ý ***@***.***>
Date: Monday, October 23, 2023 at 9:20 PM
To: usnistgov/frvt ***@***.***>
Cc: Subscribed ***@***.***>
Subject: Re: [usnistgov/frvt] [FRVT 1:1] Can I edit the header file? (Issue #113)

Thanks for your answer, in file cpp, matching is random. Can I modify it.
[Image removed by sender. Screenshot from 2023-10-24 08-15-49]<https://user-images.githubusercontent.com/55545444/277520535-cc8014f3-000b-4354-b6f3-d7199a5f53ae.png>
What are editable files?, util.h, util.cpp,validate11.cpp ?

—
Reply to this email directly, view it on GitHub<https://github.com/usnistgov/frvt/issues/113#issuecomment-1776312895>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ADVZFNJ4MBWQCZMGHTDRHNDYA4JVNAVCNFSM6AAAAAA6LOCC42VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONZWGMYTEOBZGU>.
You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>


![Screenshot from 2023-11-30 09-19-34](https://github.com/usnistgov/frvt/assets/149577702/45ea4c75-e23d-4a82-b1d4-a8ac9a0c50e7)
some members ?  can I add some attributes in nullimplfrvt11.h  @mlngan @paddygr @yhbembedding
GitHub issue: What is include in the 1:N finalization time limited?
State: closed

Hi there,

In the FRVT 1:N time limits ([5.1. Time limits](https://pages.nist.gov/frvt/api/FRVT_IREX_ongoing_1N_api.pdf)), I'm wondering to know what is included in the **1:N finalization time limited** (mark with red box)?

<img src="https://github.com/usnistgov/frvt/assets/133086055/9875eba9-52a5-449e-97a1-2a8d797343dc" width="70%" >

In detail, which following is 1:N finalization time limited (mark with red box) represented?

A. Feature extraction execution time (mark with blue box) + Enrolled database execution time (mark with green box)

B. Only include enrolled database execution time (mark with green box)

<img src="https://github.com/usnistgov/frvt/assets/133086055/6932e77a-9136-4429-9bad-5faacad643c9" width="70%" >




Answer is B.  Finalization takes N templates and produces a searchable data structure.

From: LionLion888 ***@***.***>
Date: Wednesday, October 18, 2023 at 2:13 AM
To: usnistgov/frvt ***@***.***>
Cc: Subscribed ***@***.***>
Subject: [usnistgov/frvt] What is include in the 1:N finalization time limited? (Issue #112)

Hi there,

In the FRVT 1:N time limits (5.1. Time limits<https://pages.nist.gov/frvt/api/FRVT_IREX_ongoing_1N_api.pdf>), I'm wondering to know what is included in the 1:N finalization time limited (mark with red box)?
[https://user-images.githubusercontent.com/133086055/276111081-9875eba9-52a5-449e-97a1-2a8d797343dc.png]<https://user-images.githubusercontent.com/133086055/276111081-9875eba9-52a5-449e-97a1-2a8d797343dc.png>

In detail, which following is 1:N finalization time limited (mark with red box) represented?

A. Feature extraction execution time (mark with blue box) + Enrolled database execution time (mark with green box)

B. Only include enrolled database execution time (mark with green box)
[https://user-images.githubusercontent.com/133086055/276111122-6932e77a-9136-4429-9bad-5faacad643c9.png]<https://user-images.githubusercontent.com/133086055/276111122-6932e77a-9136-4429-9bad-5faacad643c9.png>

—
Reply to this email directly, view it on GitHub<https://github.com/usnistgov/frvt/issues/112>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ADVZFNOPF3GG4OCX44TCTYLX75XOZAVCNFSM6AAAAAA6E6D7K2VHI2DSMVQWIX3LMV43ASLTON2WKOZRHE2DQOJSGEYDIMA>.
You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>


Hi paddygr,

I got it! Thanks for your response.
GitHub issue: Time Limits clarification for creation of M Templates from a single image
State: closed

Hi everyone!

In **Table 6** from [this document](https://pages.nist.gov/frvt/api/FRVT_ongoing_11_api.pdf), we can see that there is a single image and if there are M faces which our algorithm detects, then we will return M templates in the output of `createFaceTemplate` API call.

However as per **Table 1** from the same document, we can see that feature extraction should only take 1500 ms per image. And only if there are `K` images of a single person, the time limit would be scaled up by that factor. 

Am I correct in understanding following scenarios?

- We detect 5 faces in an image. Do we get 1500 ms to perform feature extraction of all the five faces **OR** do we get 1500 $\times$ 5 = 7500 ms
- We are given 5 images and we detect 1 face in each image. Will we get 1500 $\times$ 5 = 7500 ms to do the feature extraction?

Thanks,
Vinayak.

All the images you are talking about have just a single face even if you detect thousands.
You get 1500ms per image (which is one face).

In `4.4.4` of the API document, we noticed the following

![image](https://github.com/usnistgov/frvt/assets/45599762/f9e6e16f-3680-4e2d-8fdb-33d418a7f4ec)

So, we do have images which have more than one face in them right?

Thanks,
Vinayak.

Yeah there is just one foreground face

Hi @xsacha 

Agreed that there is one foreground face, but since we're not aware of which face is foreground or background, we can return M templates and then as mentioned in points 3 and 4 of section 4.4.4, matching could be done across all sets of enrollment and verification templates right?

For that we need to send multiple face templates per image and for that I was asking will we get more time or do we still have to abide to 1500 ms for getting the templates for all the faces from a single image?

Thanks,
Vinayak.

We are only conducting formal timing runs on single-person (single template) images.  While we are not enforcing timing limits for multi-person (multiple templates) images, please implement something sensible (e.g., 1500 ms * the number of faces detected would be OK).

Thanks for the clarification @mlngan 

This really helps!
GitHub issue: Inaccurate Feature Length Calculation Due to This Err
State: open

https://github.com/usnistgov/frvt/blob/ec6188c89acffe71c68cdd41c7ffa2cb64a33a7e/1N/src/testdriver/validate1N.cpp#L62C5-L62C5
The current code contains an error that leads to inaccurate calculation of feature length, affecting the overall accuracy of the finall result.

Can you please provide more details/clarification on the error that you're encountering?

Hello, @mkwml do you know which line is causing the errors? I am facing the same issue currently as well.
GitHub issue: Additional supporting library compilation on different hardware.
State: open

Hi
My package has been built successfully and it needs additional libraries which i have put in lib/ directory.
I have doubt that do we need to compile additional library on the hardware mentioned by NIST like INTEL XEON since package and supporting library is compile on different x86 hardware but with same .iso image.
GitHub issue: run_validate_ae.sh throws warnings from the sanityCheck
State: closed

During the validation process I'm getting the following warnings:
```
Sanity checking validation output 
validation/estimateAge_1.log - 
[WARNING] The following entries in .log generated non-successful return codes:
sed: can't read validation/.log: No such file or directory
[DONE]
validation/estimateAge_2.log - [DONE]
validation/verifyAge.log - 
[WARNING] The following entries in .log generated non-successful return codes:
sed: can't read validation/.log: No such file or directory
[DONE]
```

It looks like `$action` variable is not defined here: https://github.com/usnistgov/frvt/blob/master/age-estimation/run_validate_ae.sh#L50 and the same issue is present here: https://github.com/usnistgov/frvt/blob/master/age-estimation/run_validate_ae.sh#L42

It has been corrected, please update the file and thank you.
GitHub issue: Does 1:1 Face test on images rotated to 90 or 180 degree?
State: closed

I saw 2 cases in common/faces folder, rotated_90.ppm and rotated_180.ppm.

Does it mean that there are some rotated image will be test in 1:1 protocol?

We do not intend on testing with rotated images in our actual test.  HOWEVER, these edge cases were put into the validation package to test that your algorithm does not crash in the event such unexpected cases are encountered.

@mlngan and what about 1:N? Does it have rotated (90, 180) images in validation datasets?

Yes.  1:1 and 1:N share a number of images in the validation set.

> Yes. 1:1 and 1:N share a number of images in the validation set.

Thanks. And what about the _evaluation_ 1:N dataset? Should we expect rotated images there?

We do not intend on testing with rotated images in the actual 1:N test.
GitHub issue: misssed "faceunknown" mark at enroll_1N_multiface.txt
State: open

in frvt/1N/input/face/enroll_1N_multiface.txt,
```
21 ../common/images/face/S008-02-t10_01.ppm faceunknown ../common/images/face/S008-04-t10_01.ppm faceunknown ../common/images/face/S008-06-t10_01.ppm
```

It's not important... but take note
Thanks
GitHub issue: `<` wrong syntax in utils.h
State: closed

https://github.com/usnistgov/frvt/blob/b73285b59ab6b6b6b00a359ccd87bf06f7cba049/common/src/include/util.h#L122
https://github.com/usnistgov/frvt/blob/b73285b59ab6b6b6b00a359ccd87bf06f7cba049/common/src/include/util.h#L123
There are unexpected characters `<` in util.h leading fail.

<img width="1020" alt="截圖 2023-08-23 16 38 28" src="https://github.com/usnistgov/frvt/assets/66769915/690ed0f7-0630-4b73-bfd3-db14716e5e83">

fixed the issue and file is updated, thank you.
GitHub issue: Undefined enums
State: closed

The enums Action::EstimateAge and Action::VerifyAge in "common/src/util.cpp" are not defined in "common/src/include/util.h", and which causes a compile error.

Could you take care of the issue?

Updated the file, please try again.  Thank you.

It still doesn't work. There are two "<" symbols in lines 122-123 that need to be removed.

file is fixed, try again thank you.
GitHub issue: Is there any memory limit in 1:N  Identification?
State: closed

I am working on FRVT 1:N, and my query is : is there any memory limit in 1:N  Identification?

Per Section 4 of the [General Evaluation Specifications](https://pages.nist.gov/frvt/api/FRVT_common.pdf), each evaluation machine has at least 128 GB of memory. We anticipate that 16 processes can be run without time slicing, though NIST will handle all multiprocessing work via fork().  Hope this information helps.
GitHub issue: What kind of vector is in the template? 1:N
State: closed

Hi there,

When **createTemplate(TemplateRole=Enrollment_1N)**, the implementation will return a template to the calling application. 

I am wondering to know what kind of vector is in the template? Is it a **feature** from the face detection or it can be a **similarity score** produce by the face recognition as well. 

<img src=https://imgur.com/PZXbC3i.png width=60% />

Besides, if the vector in the template can be a similarity score, does that mean the **Template Generation** will deal with face detection and face recognition, also the total of the face detection and face recognition execution time need to within 1.5 sec.

<img src=https://imgur.com/XIoYV78.png width=60% />

createTemplate produces feature data from one image.   The function takes one image, so there is nothing to match against, so there is no recognition, no similarity score.

From: LionLion888 ***@***.***>
Date: Tuesday, August 8, 2023 at 6:18 AM
To: usnistgov/frvt ***@***.***>
Cc: Subscribed ***@***.***>
Subject: [usnistgov/frvt] What kind of vector is in the template? 1:N (Issue #102)

Hi there,

When createTemplate(TemplateRole=Enrollment_1N), the implementation will return a template to the calling application.

I am wondering to know what kind of vector is in the template? Is it a feature from the face detection or it can be a similarity score produce by the face recognition as well.

[Image removed by sender.]<https://camo.githubusercontent.com/3ca920d1ec7bce0560c019b1465b782d8c384f2cdecb238e67c77556b444f8bd/68747470733a2f2f696d6775722e636f6d2f505a58624333692e706e67>

Besides, if the vector in the template can be a similarity score, does that mean the Template Generation will deal with face detection and face recognition, also the total of the face detection and face recognition execution time need to within 1.5 sec.

[Image removed by sender.]<https://camo.githubusercontent.com/dcf20b311188703d4941006761e76f94684dae44375932d1d61d1542e027b360/68747470733a2f2f696d6775722e636f6d2f58496f595637382e706e67>

—
Reply to this email directly, view it on GitHub<https://github.com/usnistgov/frvt/issues/102>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ADVZFNNNN4T2HF7RABZ5BC3XUIHANANCNFSM6AAAAAA3IHVDJA>.
You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>
GitHub issue: How many images does search dataset have? 1:N
State: closed

Hi,

In the 1:N identify project, the enrollment dataset will get 1 million images as input data, if face detection failed, it shall still return a blank template and continue to go through the next enroll image. 

For the search dataset, I'm wondering there would be how many images? In the following picture, the search photo seems to be only one image. (Wanted to make sure it wold be **one person with one image** or **one person with multi images**.)

![image](https://github.com/usnistgov/frvt/assets/133086055/90187b80-9501-41f1-b93e-0ef969a7b301)



The search/probe could be any of the following
1. one image of one person (Function A)
2. multiple images of one person (Function A)
3. one image with multiple people in it (Function B)

**Function A:** 
FRVT::ReturnStatus
    createFaceTemplate(
        const std::vector<FRVT::Image> &faces,
        FRVT::TemplateRole role,
        std::vector<uint8_t> &templ,
        std::vector<FRVT::EyePair> &eyeCoordinates)

**Function B:** 
FRVT::ReturnStatus
    createFaceTemplate(
        const FRVT::Image &image,
        FRVT::TemplateRole role,
        std::vector<std::vector<uint8_t>> &templ,
        std::vector<FRVT::EyePair> &eyeCoordinates)


Thanks for your reply!

Look at the N values at the top of the columns here:
https://pages.nist.gov/frvt/html/frvt1N.html



From: LionLion888 ***@***.***>
Date: Tuesday, August 8, 2023 at 2:16 AM
To: usnistgov/frvt ***@***.***>
Cc: Subscribed ***@***.***>
Subject: Re: [usnistgov/frvt] How many images does search dataset have? 1:N (Issue #101)

Thanks for your reply!

—
Reply to this email directly, view it on GitHub<https://github.com/usnistgov/frvt/issues/101#issuecomment-1668971736>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ADVZFNP5QKGF34KEOZMPDNTXUHKSRANCNFSM6AAAAAA3GXIAFQ>.
You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>
GitHub issue: The twins project exist old type "Multiface"
State: closed

In the twins project, the program compilation fails due to the old type "Multiface" in the API parameters, please help to solve this problem.

The Multiface structure has been removed from the null implementation.
GitHub issue: Error about frvt11 test
State: closed

I prepared  my environment and tried to test frvt11 ,and  got this error.

```
_~/frvt0501/11/src/testdriver/validate11.cpp:96:29: error: ‘using element_type = class FRVT_11::Interface’ {aka ‘class FRVT_11::Interface’} has no member named ‘createTemplate’; did you mean ‘createFaceTemplate’?
 96 |         auto ret = implPtr->createTemplate(faces, role, templ, eyes);
      |                             ^~~~~~~~~~~~~~
      |                             createFaceTemplate
```
I found  ```src/nullImpl//nullimplfrvt11.h```  function name changed to ```createFaceTemplate``` and ```createIrisTemplate```
how should I solve it ?


Apologies for this - a fixed has been pushed.
GitHub issue: Questions about the latest code structure changes for the FRVT
State: closed

Hi 

I have a question about the latest master code:

Is the latest master branch code structure of FRVT project changed as follows?
yzx@7750:~/frvt-master-0410/1N$ tree .
.
└── src
    └── include
        └── frvt1N.h

2 directories, 1 file
yzx@7750:~/frvt-master-0410/1N$

**BTW:**
On February 15th the code structure looked like this:
yzx@7750:~/frvt-master-0215/1N$ tree .
.
├── CMakeLists.txt
├── input
│   ├── enroll_1N_multiface.txt
│   ├── enroll_1N_short.txt
│   ├── enroll_1N.txt
│   ├── insert.txt
│   ├── search_1N_multiperson.txt
│   └── search_1N.txt
├── README.md
├── run_validate_1N.sh
├── scripts
│   ├── build_null_impl.sh
│   ├── compile_and_link.sh
│   └── run_testdriver.sh
└── src
    ├── include
    │   └── frvt1N.h
    ├── nullImpl
    │   ├── CMakeLists.txt
    │   ├── nullimplfrvt1N.cpp
    │   └── nullimplfrvt1N.h
    └── testdriver
        ├── CMakeLists.txt
        └── validate1N.cpp

6 directories, 18 files
yzx@7750:~/frvt-master-0215/1N$
GitHub issue: There are two questions about whether the templateID will be duplicated in the input folder in the 1:N test
State: closed

Hi,
We are preparing a NIST 1:N submission and are facing some issues:
1. Will the templateID of the enroll_1N_multiface.txt under the input folder be duplicated in the real test?
    For example:
    It currently looks like this:
    1 .. /common/images/S001-01-t10_01.ppm UNKNOWN .. /common/images/S001-03-t10_01.ppm UNKNOWN .. /common/images/S001-05-t10_01.ppm UNKNOWN 
    21 .. /common/images/S008-02-t10_01.ppm UNKNOWN .. /common/images/S008-04-t10_01.ppm UNKNOWN .. /common/images/S008-06-t10_01.ppm
    45 .. /common/images/S023-01-t10_01.ppm UNKNOWN .. /common/images/S023-03-t10_01.ppm UNKNOWN

    Will this happen？:
    1 .. /common/images/S001-01-t10_01.ppm UNKNOWN .. /common/images/S001-03-t10_01.ppm UNKNOWN .. /common/images/S001-05-t10_01.ppm UNKNOWN 
   21 .. /common/images/S008-02-t10_01.ppm UNKNOWN .. /common/images/S008-04-t10_01.ppm UNKNOWN .. /common/images/S008-06-t10_01.ppm
   21 .. /common/images/S001-07-t10_01.ppm UNKNOWN .. /common/images/S001-08-t10_02.ppm UNKNOWN .. /common/images/S001-09-t10_01.ppm UNKNOWN
   45 .. /common/images/S023-01-t10_01.ppm UNKNOWN .. /common/images/S023-03-t10_01.ppm UNKNOWN

2. Will the templateID in the insert.txt under the input folder coincide with the templateID in the enroll_1N.txt?

Thank you.
Looking forward to your reply.

1. No.  The enrollment template IDs will always be unique.
2. No.  Template IDs that get inserted will not coincide with what's already enrolled in the gallery. 

Thanks for your quick reply！
GitHub issue: Pad track: evasion
State: closed

Dear @mlngan 

I am writing to ask if there is mandatory to implement evasion for the new pad challenge. If case it is optional, should we return not implemented error status to not be considered for that track?

Best,
/m

Per the API and evaluation plan, it is not required to implement both PAD functions.  For the function that you choose not to implement, please return ReturnCode::NotImplemented.

Thanks for the quick reply. In that case, I believe that the algorithm is not even considered for the other evaluation type, right? 
Best,
/m

That is correct.
GitHub issue: run_validation_11 not found
State: closed

Hi

I don't see any file with name run_validation_11.sh like README.md in 11.
Please review that.

Thanks,

Humm? I'm not from the NIST but on the master branch you have the file you're looking for : https://github.com/usnistgov/frvt/blob/master/11/run_validate_11.sh
Best regards

There was a typo in the README.md file.  We've fixed the name of the script.