Repository: RasaHQ/rasa
Pull request: remake: Tensorlfow cpp warn log supression for pytest
remake of #8702 / seems there was some problem with my poetry version that messed up dependencies

**Proposed changes**:
- fixes #7762 
- added pytest-env dependency
- added TF_CPP_MIN_LOG_LEVEL=2 as default to pytest.ini to suppress tensorflow C++ code warnings, info, and debug by default


**Status (please check what you already did)**:
- [ ] added some tests for the functionality
- [ ] updated the documentation
- [ ] updated the changelog (please check [changelog](https://github.com/RasaHQ/rasa/tree/main/changelog) for instructions)
- [ ] reformat files using `black` (please check [Readme](https://github.com/RasaHQ/rasa#code-style) for instructions)


Problem statement:
Suppress noisy tensorflow logs on linux and Windows
**Rasa version**:
2.2.4
**Python version**:
3.7.2
**Operating system** (windows, osx, ...):
Windows, Linux

**Issue**:

It seems that the hack to suppress noisy tensorflow logs is no longer working for Windows or Linux. This confuses users unnecessarily.

**Error (including full traceback)**:
example:
```
2021-01-15 12:12:22.059483: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'libcudart.so.10.1'; dlerror: libcudart.so.10.1: cannot open shared object file: No such file or directory
```

**Definition of Done**:

- [ ] determine the cause of Tensorflow warnings when using `rasa` in non-debug mode & document
- [ ] fix the current issue (that warnings get shown)

Hints from issue discussion:
Relates to: https://github.com/RasaHQ/rasa/issues/7738
Based on [this Slack discussion](https://rasa-hq.slack.com/archives/C8LEHMTAM/p1617097796180900?thread_ts=1617090758.172800&cid=C8LEHMTAM), we are going to move this to Enable
This particular warning doesn't seem to be just a nuisance.

Found [a tensorflow issue](https://github.com/tensorflow/tensorflow/issues/38578) on this and it seems like the dependency (at least the specific version) is missing and in the output it also says:

**Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU [...] Skipping registering GPU devices...**
```
[...]
2020-04-15 20:58:07.205269: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] 
Could not load dynamic library 'libcudart.so.10.1'; dlerror: libcudart.so.10.1: cannot open shared 
object file: No such file or directory
[...]
2020-04-15 20:58:07.219842: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1598] 
Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are 
installed properly if you would like to use GPU. Follow the guide at 
https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for 
your platform. Skipping registering GPU devices...
```


@koernerfelicia, where is this warning log from? Also, looks like you have some PR open to change logging level - what should happen with that? And, was the GPU in fact not usable?
@twerkmeister I think in many cases (and all that I am wanting to address) it *is* just a nuisance -- we get these error messages even when there's no GPU available, hence these libraries don't need to be installed:

See CI logs [here](https://github.com/RasaHQ/rasa/pull/7990/checks?check_run_id=2149382304#step:12:54):
`Ignore above cudart dlerror if you do not have a GPU set up on your machine.`

The libraries it's looking for are CUDA-related libraries, which you *do* need for a GPU setup, but *don't* for a purely CPU setup. I think we're seeing this because you used to have to install a specific version of `tensorflow` to be able to use a GPU ([`tensorflow-gpu`](https://www.tensorflow.org/install/gpu#older_versions_of_tensorflow)), and now even the "standard" `tensorflow` is GPU-compatible. My guess is with the switch to the single tensorflow with GPU suppport they decided to just look for the CUDA libraries either way, complain if they're not found, and let the user decide whether they need the library or not (whereas before, I imagine `tensorflow-gpu` looked for them, and `tensorflow` didn't).

> And, was the GPU in fact not usable?

I haven't observed this issue with a GPU, as I have always had the CUDA libraries installed. I can't imagine that the GPU would be usable without the CUDA libraries (as far as I know, non-CUDA architectures are not supported by `tensorflow`, and I'm not sure we have access to these types of architectures on google compute anyways).


> where is this warning log from

As far as I can tell, the logs are appearing for Windows and Linux users. They can be observed in our CI logs (see link above, for example).

> like you have some PR open to change logging level - what should happen with that

The PR is only useful as a record of what I tried to fix the issue. None of these fixes worked.

Happy to jump on a call to discuss any of this!
Slack discussion [here](https://rasa-hq.slack.com/archives/C8LEHMTAM/p1610714890070700?thread_ts=1610558144.253400&cid=C8LEHMTAM) (maybe not worth reading -- the gist of it is that we *used* to suppress these according to Daksh, and that neither of us were seeing this on our laptops)
Thanks for all the context! Makes sense. Will see if there's some setting to let tensorflow go for cpu directly if there's no gpu anyway!
Also not getting this warning on my laptop either, seems there might be a slight difference between local and ci environment
What OS? I think it might just be Linux and Windows
locally I am on mac os...

I was just about to benchmark another issue on a google cloud machines using a template from an earlier research machine and got the same warning when using rasa. Turns out that cloud machine only had cuda 10.0 installed, but our tf version looks for cuda 10.1. Seems last version that used cuda 10.0 was 2.0. Did we just upgrade tf recently?


Not *that* recently, think around fall of last year? Not sure exactly, I think this was before I started (Oct). Can find this out for you, though.

CUDA and tensorflow can be a bit tricky -- the standard pip install isn't always compatible with just any old CUDA. You can compile from source for a specific TF/CUDA combination, but this is also a bit of a pain, so I'd try to get an image that has a a compatible tensorflow+CUDA combo already installed. This also means you'll have to specifically use that tensorflow in a virtualenv or not use a virtualenv at all. Happy to jump on a call to explain this, we've had various issues with this. There are some details [here](https://www.notion.so/rasa/Running-Experiments-on-Google-Cloud-41c50e7e7b894d69acf08b25e82ef898#2471f799aa3b4c6fbc3cd7b30f2e266d)

Commit message: Added pytest-env dependency, added tensorlfow warn log supression for all pytest invocations

Patch to review:
diff --git a/poetry.lock b/poetry.lock
index 8a253fce13e8..ff1219fe94b4 100644
--- a/poetry.lock
+++ b/poetry.lock
@@ -2097,6 +2097,17 @@ pytest = ">=4.6"
 [package.extras]
 testing = ["fields", "hunter", "process-tests (==2.0.2)", "six", "pytest-xdist", "virtualenv"]
 
+[[package]]
+name = "pytest-env"
+version = "0.6.2"
+description = "py.test plugin that allows you to add environment variables."
+category = "dev"
+optional = false
+python-versions = "*"
+
+[package.dependencies]
+pytest = ">=2.6.0"
+
 [[package]]
 name = "pytest-forked"
 version = "1.3.0"
@@ -3280,7 +3291,7 @@ transformers = ["transformers"]
 [metadata]
 lock-version = "1.1"
 python-versions = ">=3.6,<3.9"
-content-hash = "aa984cf2e02816e60bcaa5bca471ad6f42167526dca33e02f8967d9ee2c9e72c"
+content-hash = "ce3dc37582031a689645cbe61470309dc6071f6b9dc62d30019b2752f44d30b0"
 
 [metadata.files]
 absl-py = [
@@ -4648,6 +4659,9 @@ pytest-cov = [
     {file = "pytest-cov-2.11.1.tar.gz", hash = "sha256:359952d9d39b9f822d9d29324483e7ba04a3a17dd7d05aa6beb7ea01e359e5f7"},
     {file = "pytest_cov-2.11.1-py2.py3-none-any.whl", hash = "sha256:bdb9fdb0b85a7cc825269a4c56b48ccaa5c7e365054b6038772c32ddcdc969da"},
 ]
+pytest-env = [
+    {file = "pytest-env-0.6.2.tar.gz", hash = "sha256:7e94956aef7f2764f3c147d216ce066bf6c42948bb9e293169b1b1c880a580c2"},
+]
 pytest-forked = [
     {file = "pytest-forked-1.3.0.tar.gz", hash = "sha256:6aa9ac7e00ad1a539c41bec6d21011332de671e938c7637378ec9710204e37ca"},
     {file = "pytest_forked-1.3.0-py2.py3-none-any.whl", hash = "sha256:dc4147784048e70ef5d437951728825a131b81714b398d5d52f17c7c144d8815"},
diff --git a/pyproject.toml b/pyproject.toml
index c8cd3ed1a97e..28a0221740d7 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -152,6 +152,7 @@ bandit = "^1.6.3"
 typing-extensions = "^3.7.4"
 memory-profiler = "^0.58.0"
 psutil = "^5.8.0"
+pytest-env = "^0.6.2"
 
 [tool.poetry.extras]
 spacy = [ "spacy",]
diff --git a/pytest.ini b/pytest.ini
index 99cb56ba85e1..6baa61319d5c 100644
--- a/pytest.ini
+++ b/pytest.ini
@@ -20,3 +20,7 @@ markers =
 # see https://pypi.org/project/pytest-timeout/
 timeout = 60
 timeout_func_only = true
+
+# the D: (read: default) prefix specifies that existing env vars are not overwritten
+env =
+    D:TF_CPP_MIN_LOG_LEVEL=2


Code review comments:
diff_hunk: @@ -152,6 +152,7 @@ bandit = "^1.6.3"
 typing-extensions = "^3.7.4"
 memory-profiler = "^0.58.0"
 psutil = "^5.8.0"
+pytest-env = "^0.6.2"
path: pyproject.toml
text: @user1:
I'm a bit hesitant to add a dependency which wasn't updated since 2017 to our codebase even if it's just a dev dependency. How about using `pytest` [hooks](https://docs.pytest.org/en/latest/how-to/writing_hook_functions.html) for it or editing our `Makefile`?

@author:
Fair point about the updates. At the same time its a 50 code line plugin. And I think it does what it is supposed to do in a straightforward and discoverable manner, setting the env var for all pytest runs.

I think using hooks to set env variables obfuscates things. Make file could be adjusted, there we just need to put it at all invocations, which is 4 so far. There's also another variable - OMP_NUM_THREADS=1 that is set there already. So either we can add the log var there or pull the OMP into the pytest env.

@user1:
Let's do the Makefile then + adding a comment why we need it.