GitHub issue: Query on Final Submission - FRVT11
State: closed

I have successfully generated the tar.gz file for submission, and manual analysis of the validation package shows above 90% F1, so I guess I am all ready for submission for FRVT11 :) 

My approach to FRVT11 is to embed my python codes and model in C++, hence in my my CMakeList in ./src/nullImpl directory, I have called:
```
find_package(PythonLibs REQUIRED) 
target_link_libraries (frvt_11_COMPANY_000 ${PYTHON_LIBRARIES})
```
plus my include directories and so on. In addition, I have to put my tensorflow model file in ./frvt/11 main directory in order to run run_validate_11.sh

My question is this: Can I put all these python packages, include directories and python model files in the ./frvt/11/lib folder?
For instance in the lib folder, I have:
-  libfrvt_11_COMPANY_000.so (file)
- python3.8_include (directory)
- python3.8_libraries (directory)
- tensorflow_model (directory)

Appreciate the assistance, and thanks in advance!


You just need your python libs. There is no need to include source files (includes).
I recommend spin up another clean VM, extract your package there and attempt to run it. If it works, you're all sweet.

Pardon me, but may I also ask what files are required in the ./config folder? I did not require any files in the ./config folder in order to run my validation.

Another clarification: When I compile my codes with ./scripts/build_null_impl.sh, should my python3.8_libraries (directory) be already in the /lib folder? Because I only added it in after completion. Initially the python libraries should be called from my local directory. 

Well it measures the size of the config directory as your model size so it assumes your models go there. However, you could just as easily put them inside your libs.

Putting your libs in after will be fine but remember to test it.

During test time by FRVT when I have sent my tar.gz file over, would the .so file be run automatically based on the python libraries and models already in the ./lib folder? Would there be any manual adjustments on these files to different directories, or Python installation made by NIST personnel?  

I am asking because I understand that in the test environment Python would not have been installed. I am not sure how would it be set up and tested, and I hope my submission works in the NIST environment.

Imagine they run:
LD_LIBRARY_PATH=/path/to/submission/lib testApp

You should be able to test this yourself in a VM (not the one you used to build) using the script that is run during the submission file creation.

If this isn't coming across right, an easy alternative is: just create the submission on a clean VM 

They mention what OS image is used in the API document. Assume a clean install of that ISO.

You will need to set any Python-related environment variables (e.g., PYTHONPATH, etc.) internally within your code.  Consider putting your Python libraries in the configuration directory, which is passed in as a parameter to initialize().  If you want to put your python libraries in ./lib, you may assume that ./lib and ./config are at the same level (i.e., you can access ./lib via configDir/../lib.

I am understanding that LD_LIBRARY_PATH is for directories containing .so files, hence I should only be putting my libpython38.so etc files into the ./lib folder.

As for installed Python packages (pip install etc), I should be setting the PYTHONPATH variable. My question would be: where can I insert 'export PYTHONPATH'? Is it possible to edit `run_validate_11.sh`? But the submission package does not include `run_validate_11.sh` right?

Or perhaps can I edit the codes in `nullimplfrvt11.cpp` to include:
```
  Py_Initialize();
  PyRun_SimpleString("import sys");
  PyRun_SimpleString("sys.path.append("/Path/to/Python/Package/in/config");
```
Thanks for your assistance!

You can run this code inside your library implementation.

I understand that Ubuntu 20.04.3 LTS comes with the Python 3.8 installed. Can I assume that NIST test environment also uses a clean Ubuntu 20.04.3 LTS with Python 3.8 installed? 

Yes, you may assume that Python 3.8 is installed in our test environment.

I need to:
```
`export PYTHONPATH=$(pwd)/lib/python3.8_libraries`
`export TORCH_HUB=$(pwd)/lib/python3.8_libraries`
```

But when I run:
```
NullImplFRVT11::initialize(const std::string &configDir)
{
        Py_Initialize();
        PyRun_SimpleString("import os");
        PyRun_SimpleString("os.system('export PYTHONPATH=$(pwd)/lib/python3.8_libraries')");
        PyRun_SimpleString("os.system('export TORCH_HUB=$(pwd)/lib/python3.8_libraries')");

    return ReturnStatus(ReturnCode::Success);
}
```

It does not work.  What is the exact codes I can run to export PYTHONPATH and TORCH_HUB from nullimplfrvt11.cpp?

Thank you!


It's probably just due to your $(pwd) being different than you expect.
The config dir sounds like a more reasonable location, since that is passed in directly to the initialize function.

What happens if you use:

```C++
#include <dlfcn.h>

std::string GetMyPath()
{
    Dl_info dl_info;
    dladdr((void *)GetMyPath, &dl_info);
    return dl_info.dli_fname;
}
ReturnStatus NullImplFRVT11::initialize(const std::string &configDir)
{
    // Acquire path
    auto path = GetMyPath();
    auto pythonPath = path.substr(0, path.find_last_of('/') + 1) + "python3.8_libraries";
    // Set global vars, which will be inherited by forked processes
    setenv("PYTHONPATH", pythonPath.c_str(), 1);
    setenv("TORCH_HUB", pythonPath.c_str(), 1);

    return ReturnStatus(ReturnCode::Success);
}
```

Hi @xsacha, I have tried this and it does not work. 

Typically I just do:

```
`export PYTHONPATH=$(pwd)/lib/python3.8_libraries`
`export TORCH_HUB=$(pwd)/lib/python3.8_libraries`
```
In the run_validate_11.sh file.


But @mlngan said this is not allowed for submission


Yeah it wouldn't make sense for them to modify their environment for specific users.

How about printing out the location you get from $(pwd)/lib/python3.8_libraries to ensure it is what you expect.

Keep in mind that might not work in NISTs tests because they could run it from any directory they want -- hence the configDir being provided.

The code I posted above works fine for me but I am not using Python so it's possible the Python environment has already been set up somewhere in your code with a different environment.

printing $(pwd)/lib/python3.8_libraries does point to frvt/11/lib/python3.8_libraries, also even from the nullimplfrvt11.cpp file.

The strange thing is that when I insert 
```
`export PYTHONPATH=$(pwd)/lib/python3.8_libraries`
`export TORCH_HUB=$(pwd)/lib/python3.8_libraries`
```
In the ~/.bashrc file, the code does not work. BUT when I insert the code in the run_validate_11.sh file, it works!

For the bashrc file, you have to load a new bash shell for it to get that file to run and it is only set within that shell. Just like running export, it isn't a workable solution here.

For the Python code you were running:
```C++
        PyRun_SimpleString("os.system('export PYTHONPATH=$(pwd)/lib/python3.8_libraries')");
```
this runs the code in a separate shell (outside of your application), so it won't affect the environment of your C++ or Python NIST process whatsoever.

As for why the code I posted didn't work: unsure. It is working on my machine :) Only thing I can think of is to make sure it is run before you run PyInitialize, in case that is preserving current env and make sure path is right.

By the way for your codes posted just now, are u assuming that my python3.8_libraries are in the config directory or the lib directory? More specifically,

`auto pythonPath = path + "/python3.8_libraries";`

or

`auto pythonPath = path + "/lib/python3.8_libraries";`

Thank you!


It's probably easier if they are in config, but I am assuming they are in lib.
You can print out path to check which directory it is:
`printf("%s\n", path.c_str());`

printf("%s\n", path.c_str()) gives me:

/Documents/frvt/11/lib/libfrvt_11_COMPANY_000.so

I think it should point to /Documents/frvt/11/lib instead right?

Yes, sorry.

You can use:
auto pythonPath = path.substr(0, path.find_last_of('/') + 1) + "python3.8_libraries";

Thank you so much! <3 It worked beautifully!

No problems. If you put the libraries in the 'config' path instead, you can ditch that whole GetMyPath() and just put in:
auto pythonPath = configDir + "/python3.8_libraries";

That's the suggested method from NIST and why they provide that path.

I realized I have been using Ubuntu-20.04.3 Desktop installed from the Ubuntu archives instead of https://nigos.nist.gov/evaluations/ubuntu-20.04.3-live-server-amd64.iso when I ran my tests, hence I ran into some issues during submissions with testing over at NIST side.

During NIST import of openCV, i encountered this error:

```
Traceback (most recent call last):
  File autentika_000/lib/python3.8_libraries/extract_template.py", line 1, in <module> import cv2
  File autentika_000/lib/python3.8_libraries/cv2/__init__.py", line 181, in <module> bootstrap()
  File autentika_000/lib/python3.8_libraries/cv2/__init__.py", line 153, in bootstrap native_module = importlib.import_module("cv2")
 File autentika_000/lib/python3.8_libraries/importlib/__init__.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level)
ImportError: libGL.so.1: cannot open shared object file: No such file or directory
```

The libGL.so.1 is located in the /usr/lib/x86_64-linux-gnu directory in my Ubuntu Desktop.

How can I move it to the /lib/python3.8_libraries directory, and have the import of libGL.so.1 detected? Does this do the trick:

`setenv("PATH", pythonPath.c_str(), 1);`




You want "LD_LIBRARY_PATH"

I guess this is a pre-compiled OpenCV? You would want an OpenCV compiled without OpenGL or image loading normally. You won't be using any of that.

Basically my python OpenCV libraries are all located in  /lib/python3.8_libraries directory. Apparently i need libGL.so.1 to import python OpenCV

setenv( "LD_LIBRARY_PATH", pythonPath.c_str(), 1); did not work

The system is still looking for the libGL.so.1 file in the /usr/lib/x86_64-linux-gnu directory

I have also tried to edit the CMakeLists.txt

```
# Build the shared libraries
set (CMAKE_SHARED_LINKER_FLAGS "-Wl,-rpath,$ORIGIN")
add_library (frvt_11_autentika_000 SHARED nullimplfrvt11.cpp)
target_link_libraries (frvt_11_autentika_000 libpython3.8.so libGL.so.1)
```

But i got the error during compilation:

```
/usr/bin/ld: cannot find -lGL
collect2: error: ld returned 1 exit status
```

How can I linked the shared object files in the /lib directory? 

Are you sure it's just libGL it's looking for? I just installed opencv-python package and noticed it has a *massive* amount of dependent libraries I can see in: site-packages/opencv_python.libs/.  It has been compiled to include absolutely everything including GUI (which provided Qt on mine).

If the dependency comes from there, it should probably be enough to stick all of the missing ones in there along with them.

On mine, the dependencies on a single Qt lib provided with opencv-python:

> ~/.local/lib/python3.10/site-packages/opencv_python.libs/libQt5XcbQpa-c112ba75.so.5.15.0:
>         linux-vdso.so.1 (0x00007ffdefd2e000)
>         libpng16-57e5e0a0.so.16.37.0 => not found
>         libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f34e7b32000)
>         libGL.so.1 => /lib/x86_64-linux-gnu/libGL.so.1 (0x00007f34e7aab000)
>         libQt5Gui-48e93776.so.5.15.0 => not found
>         libQt5Core-39545cc7.so.5.15.0 => not found
>         libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f34e7aa4000)
>         libX11-xcb-69166bdf.so.1.0.0 => not found
>         libxcb-icccm-413c9f41.so.4.0.0 => not found
>         libxcb-image-e82a276d.so.0.0.0 => not found
>         libxcb-shm-7a199f70.so.0.0.0 => not found
>         libxcb-keysyms-21015570.so.1.0.0 => not found
>         libxcb-randr-a96a5a87.so.0.1.0 => not found
>         libxcb-render-util-43ce00f5.so.0.0.0 => not found
>         libxcb-render-637b984a.so.0.0.0 => not found
>         libxcb-shape-25c2b258.so.0.0.0 => not found
>         libxcb-sync-89374f40.so.1.0.0 => not found
>         libxcb-xfixes-9be3ba6f.so.0.0.0 => not found
>         libxcb-xinerama-ae147f87.so.0.0.0 => not found
>         libxcb-xkb-9ba31ab3.so.1.0.0 => not found
>         libxcb.so.1 => /lib/x86_64-linux-gnu/libxcb.so.1 (0x00007f34e7a76000)
>         libXext.so.6 => /lib/x86_64-linux-gnu/libXext.so.6 (0x00007f34e7a5f000)
>         libX11.so.6 => /lib/x86_64-linux-gnu/libX11.so.6 (0x00007f34e791f000)
>         libSM.so.6 => /lib/x86_64-linux-gnu/libSM.so.6 (0x00007f34e7914000)
>         libICE.so.6 => /lib/x86_64-linux-gnu/libICE.so.6 (0x00007f34e78f7000)
>         libxkbcommon-x11-c65ed502.so.0.0.0 => not found
>         libxkbcommon-71ae2972.so.0.0.0 => not found
>         ...

I realise this is very painful going through Python. Tip to anyone else trying this: just use C++ :)

Instead of Python OpenCV, do you think I can use skimage or PIL to process images?

Sure, I recommend PIL if that works for you. It won't draw in the ~100 .so files that OpenCV does.

Since you aren't opening images or converting them (provided for you as bytes), you could probably get away with just numpy.