Repository: scikit-hep/pyhf

Issue:
Fix TensorFlow v2.3.0 and TensorFlow Probability v0.11.0 release incompatibilities
# Description

[TensorFlow `v2.3.0`](https://github.com/tensorflow/tensorflow/releases/tag/v2.3.0) has been released today (2020-07-27) but is breaking the tests in CI with the following:

```
    pythonLocation: /opt/hostedtoolcache/Python/3.7.8/x64
2020-07-27 21:25:53.025362: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'libcudart.so.10.1'; dlerror: libcudart.so.10.1: cannot open shared object file: No such file or directory
2020-07-27 21:25:53.025414: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
ImportError while loading conftest '/home/runner/work/pyhf/pyhf/tests/conftest.py'.
tests/conftest.py:83: in <module>
    (pyhf.tensor.tensorflow_backend(), None),
src/pyhf/tensor/__init__.py:50: in __getattr__
    e,
E   pyhf.exceptions.ImportBackendError: ('There was a problem importing TensorFlow. The tensorflow backend cannot be used.', ImportError("cannot import name 'naming' from 'tensorflow.python.autograph.core' (/opt/hostedtoolcache/Python/3.7.8/x64/lib/python3.7/site-packages/tensorflow/python/autograph/core/__init__.py)"))
##[error]Process completed with exit code 4.
```

the code snippet that is being reported here is:

https://github.com/scikit-hep/pyhf/blob/60488cd4223dcc5519605b766db1f9af91fa79fb/src/pyhf/tensor/__init__.py#L39-L51

but the actual error is

```
2020-07-27 21:25:53.025362: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'libcudart.so.10.1'; dlerror: libcudart.so.10.1: cannot open shared object file: No such file or directory
```

as with TensorFlow `v2.3.0` just importing gets you the attempt to load `libcudart.so.10.1`

```python
>>> import tensorflow as tf
2020-07-27 16:40:48.374806: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
```


# Expected Behavior

The test suite runs

# Actual Behavior

c.f. above

# Steps to Reproduce

Run the CI.

# Checklist

- [x] Run `git fetch` to get the most up to date version of `master`
- [x] Searched through existing Issues to confirm this is not a duplicate issue
- [x] Filled out the Description, Expected Behavior, Actual Behavior, and Steps to Reproduce sections above or have edited/removed them in a way that fully describes the issue



Discussion:
Though this might be something that we need to catch as just trying to import `tensorflow` (`python -c "import tensorflow as tf"`) after the dependencies are installed gives

```
2020-07-27 21:51:18.905311: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'libcudart.so.10.1'; dlerror: libcudart.so.10.1: cannot open shared object file: No such file or directory
2020-07-27 21:51:18.905345: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
```
@kratsg has pointed out that this is actually a problem with TensorFlow Probability `v0.10.X` releases as `v0.11.X` uses TF `v2.3.X`.

```
$ docker run --rm -it python:3.8-slim /bin/bash
# pip install pyhf[tensorflow]
# pip list | grep tensor
tensorboard            2.3.0
tensorboard-plugin-wit 1.7.0
tensorflow             2.3.0
tensorflow-estimator   2.3.0
tensorflow-probability 0.10.1
# python
Python 3.8.2 (default, Apr 23 2020, 14:32:57) 
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pyhf
>>> pyhf.set_backend("tensorflow")
2020-07-27 23:16:40.268209: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'libcudart.so.10.1'; dlerror: libcudart.so.10.1: cannot open shared object file: No such file or directory
2020-07-27 23:16:40.268358: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/pyhf/tensor/__init__.py", line 41, in __getattr__
    from .tensorflow_backend import tensorflow_backend
  File "/usr/local/lib/python3.8/site-packages/pyhf/tensor/tensorflow_backend.py", line 4, in <module>
    import tensorflow_probability as tfp
  File "/usr/local/lib/python3.8/site-packages/tensorflow_probability/__init__.py", line 76, in <module>
    from tensorflow_probability.python import *  # pylint: disable=wildcard-import
  File "/usr/local/lib/python3.8/site-packages/tensorflow_probability/python/__init__.py", line 24, in <module>
    from tensorflow_probability.python import experimental
  File "/usr/local/lib/python3.8/site-packages/tensorflow_probability/python/experimental/__init__.py", line 34, in <module>
    from tensorflow_probability.python.experimental import auto_batching
  File "/usr/local/lib/python3.8/site-packages/tensorflow_probability/python/experimental/auto_batching/__init__.py", line 24, in <module>
    from tensorflow_probability.python.experimental.auto_batching import frontend
  File "/usr/local/lib/python3.8/site-packages/tensorflow_probability/python/experimental/auto_batching/frontend.py", line 45, in <module>
    from tensorflow.python.autograph.core import naming
ImportError: cannot import name 'naming' from 'tensorflow.python.autograph.core' (/usr/local/lib/python3.8/site-packages/tensorflow/python/autograph/core/__init__.py)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.8/site-packages/pyhf/events.py", line 68, in register_wrapper
    result = func(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/pyhf/__init__.py", line 88, in set_backend
    backend = getattr(tensor, f"{backend:s}_backend")(**backend_kwargs)
  File "/usr/local/lib/python3.8/site-packages/pyhf/tensor/__init__.py", line 48, in __getattr__
    raise exceptions.ImportBackendError(
pyhf.exceptions.ImportBackendError: ('There was a problem importing TensorFlow. The tensorflow backend cannot be used.', ImportError("cannot import name 'naming' from 'tensorflow.python.autograph.core' (/usr/local/lib/python3.8/site-packages/tensorflow/python/autograph/core/__init__.py)"))
>>>
# pip install --upgrade "tensorflow-probability>=v0.11.0-rc1"
# pip list | grep tensor
tensorboard            2.3.0
tensorboard-plugin-wit 1.7.0
tensorflow             2.3.0
tensorflow-estimator   2.3.0
tensorflow-probability 0.11.0rc1
# python
Python 3.8.2 (default, Apr 23 2020, 14:32:57) 
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pyhf
>>> pyhf.set_backend("tensorflow")
2020-07-27 23:18:06.709873: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'libcudart.so.10.1'; dlerror: libcudart.so.10.1: cannot open shared object file: No such file or directory
2020-07-27 23:18:06.709903: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
>>> 
```
Some good news on the issues RE: SciPy and TF `v2.3.0` and TFP `v0.11.0` seen in PR #1001: https://groups.google.com/a/tensorflow.org/d/msg/discuss/wLRLe96ymFg/vlNecsSZAwAJ

TF `v2.4.0` "is slated to come out sometime around september/october." and a patch release of `v2.3.X` might be possible. :+1: Thanks to @alextp for being super responsive too.
Further related to PR #1001 are https://github.com/tensorflow/tensorflow/pull/41867 and https://github.com/tensorflow/tensorflow/issues/41902. I've gone ahead and subscribed to these so that we can keep tabs on them and act quickly.

There is additionally a discussion ongoing at https://github.com/pypa/pip/issues/8076 RE: the `pip` October 2020 dependency resolver changes that is relevant as well until a patch release of TensorFlow comes out.
What is the status of this? [`TensorFlow` 2.5.0](https://github.com/tensorflow/tensorflow/releases/tag/v2.5.0) is available now, so maybe some of the above issues are resolved. I am looking into this as I was hoping to find a setup with `pyhf` that allows testing the `TensorFlow` backend but also to use `numpy`>=1.20 for typing reasons (reference: https://github.com/alexander-held/cabinetry/pull/228). Installing `TensorFlow` 2.5.0 picks up `numpy~=1.19.2`, so it seems like this unfortunately may not be possible yet.

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -3,8 +3,8 @@
 extras_require = {
     'shellcomplete': ['click_completion'],
     'tensorflow': [
-        'tensorflow~=2.2.1',  # TensorFlow minor releases are as volatile as major
-        'tensorflow-probability~=0.10.1',
+        'tensorflow~=2.2,>=2.2.1,!=2.3.0',  # c.f. https://github.com/tensorflow/tensorflow/pull/40789
+        'tensorflow-probability~=0.10,>=0.10.1',
     ],
     'torch': ['torch~=1.8'],
     'jax': ['jax~=0.2.8', 'jaxlib~=0.1.58,!=0.1.68'],  # c.f. Issue 1501
diff --git a/src/pyhf/tensor/tensorflow_backend.py b/src/pyhf/tensor/tensorflow_backend.py
--- a/src/pyhf/tensor/tensorflow_backend.py
+++ b/src/pyhf/tensor/tensorflow_backend.py
@@ -2,6 +2,7 @@
 import logging
 import tensorflow as tf
 import tensorflow_probability as tfp
+from numpy import nan
 
 log = logging.getLogger(__name__)
 
@@ -119,7 +120,7 @@ def tile(self, tensor_in, repeats):
         """
         try:
             return tf.tile(tensor_in, repeats)
-        except tf.python.framework.errors_impl.InvalidArgumentError:
+        except tf.errors.InvalidArgumentError:
             shape = tf.shape(tensor_in).numpy().tolist()
             diff = len(repeats) - len(shape)
             if diff < 0:
@@ -426,8 +427,15 @@ def poisson_logpdf(self, n, lam):
             TensorFlow Tensor: Value of the continuous approximation to log(Poisson(n|lam))
         """
         lam = self.astensor(lam)
+        # Guard against Poisson(n=0 | lam=0)
+        # c.f. https://github.com/scikit-hep/pyhf/issues/293
+        valid_obs_given_rate = tf.logical_or(
+            tf.math.not_equal(lam, n), tf.math.not_equal(n, 0)
+        )
 
-        return tfp.distributions.Poisson(lam).log_prob(n)
+        return tf.where(
+            valid_obs_given_rate, tfp.distributions.Poisson(lam).log_prob(n), nan
+        )
 
     def poisson(self, n, lam):
         r"""
@@ -457,8 +465,17 @@ def poisson(self, n, lam):
             TensorFlow Tensor: Value of the continuous approximation to Poisson(n|lam)
         """
         lam = self.astensor(lam)
+        # Guard against Poisson(n=0 | lam=0)
+        # c.f. https://github.com/scikit-hep/pyhf/issues/293
+        valid_obs_given_rate = tf.logical_or(
+            tf.math.not_equal(lam, n), tf.math.not_equal(n, 0)
+        )
 
-        return tf.exp(tfp.distributions.Poisson(lam).log_prob(n))
+        return tf.where(
+            valid_obs_given_rate,
+            tf.exp(tfp.distributions.Poisson(lam).log_prob(n)),
+            nan,
+        )
 
     def normal_logpdf(self, x, mu, sigma):
         r"""


Test patch:
diff --git a/tests/test_tensor.py b/tests/test_tensor.py
--- a/tests/test_tensor.py
+++ b/tests/test_tensor.py
@@ -392,7 +392,7 @@ def test_tensor_tile(backend):
     ]
 
     if tb.name == 'tensorflow':
-        with pytest.raises(tf.python.framework.errors_impl.InvalidArgumentError):
+        with pytest.raises(tf.errors.InvalidArgumentError):
             tb.tile(tb.astensor([[[10, 20, 30]]]), (2, 1))