GitHub issue: Thread warning when using OpenVINO backend
State: closed

My implementation is single threaded, but it will be using OpenVINO as the inference backend, which is threaded. Upon running the validation script, I received the following warning:

```
[WARNING] We've detected that your software may be threading or using other multiprocessing techniques.  The number of 
processes detected was 22 and it should be 2.  Per the API document, implementations must run single-threaded.  In the test 
environment, there is no advantage to threading, because NIST will distribute workload across multiple blades and multiple 
processes.  We highly recommend that you fix this issue prior to submission.
```

I saw someone mentioning in the past Issues that all the main frameworks such as tensorflow, pytorch and mxnet have been known to work in NIST. 

Do you also know from past submissions if OpenVINO had any issues with NIST?

By default it will attempt to use threads so you need to tell it not to.

> By default it will attempt to use threads so you need to tell it not to.

Thanks. I will search through the OpenVINO docs to see how to tell it not use threads.

Were you able to set the max number of threads for one inference without recompiling OpenVINO?

I recompile all the frameworks anyway but they should have a method without doing so.
Usually it is based on the internal threading library they use (TBB, etc) rather than the inferencing framework itself. Being an Intel framework, I'd assume it is TBB so you can just call, off the top of my head:
```C++
#include <tbb/global_control.h>

...

tbb::global_control MAX_THREADS(tbb::global_control::max_allowed_parallelism, 1);
```

> I recompile all the frameworks anyway but they should have a method without doing so.

It seems that they do. Going over the IE c++ api, you can specify how many threads the OpenVINO CPU Plugin (aka MKLDNN) can use for inference, among others: https://docs.openvino.ai/2021.1/openvino_docs_IE_DG_supported_plugins_CPU.html

After reading that page until the end, you can find the suggested combination of parameters for single thread inference:

> **NOTE:** To disable all internal threading, use the following set of configuration parameters: KEY_CPU_THROUGHPUT_STREAMS=0, KEY_CPU_THREADS_NUM=1, KEY_CPU_BIND_THREAD=NO.

According to my stack tracing on the validation binary, it seems I got OpenVINO multi threads out of the way now, but I seem to have found that Boost and OpenBLAS are also creating threads at some point. Luckily, for the PAD implementation, I can remove the OpenBLAS dependency entirely, so that is solved. But I am still looking on how to force boost to run on a single thread.


Which part of boost is using threads for you? Normally you'd have to explicitly call some boost function that is asynchronous. There's usually a synchronous version.

> Which part of boost is using threads for you? Normally you'd have to explicitly call some boost function that is asynchronous. There's usually a synchronous version.

Oh I see now. We are explicitly creating threads with ```boost::asio::thread_pool threadPool(std::thread::hardware_concurrency());``` at model initialization.