Repository: scikit-hep/pyhf

Issue:
GitHub Actions only error: AttributeError("'NoneType' object has no attribute '_sorted_indices'")
### Summary

In PR #1274 there is a `AttributeError("'NoneType' object has no attribute '_sorted_indices'")` that is happening for Python 3.10 only for unclear reasons. This is blocking release `v0.7.0`. This is also not reproducible locally on my laptop.

### OS / Environment

```console
GitHub Actions on Python 3.10
```


### Steps to Reproduce

<!--- ...or if you have a failing CLI command paste it between the quotes below -->
```console (paste below)
$ pytest src/pyhf/tensor/jax_backend.py
```

### File Upload (optional)

_No response_

### Expected Results

For the CI to pass

### Actual Results

```pytb
=================================== FAILURES ===================================
____________ [doctest] pyhf.tensor.jax_backend.jax_backend.astensor ____________
207 
208         Convert to a JAX ndarray.
209 
210         Example:
211 
212             >>> import pyhf
213             >>> pyhf.set_backend("jax")
UNEXPECTED EXCEPTION: AttributeError("'NoneType' object has no attribute '_sorted_indices'")
Traceback (most recent call last):
  File "/Users/runner/hostedtoolcache/Python/3.10.7/x64/lib/python3.10/doctest.py", line 1350, in __run
    exec(compile(example.source, filename, "single",
  File "<doctest pyhf.tensor.jax_backend.jax_backend.astensor[1]>", line 1, in <module>
  File "/Users/runner/work/pyhf/pyhf/src/pyhf/events.py", line 161, in register_wrapper
    result = func(*args, **kwargs)
  File "/Users/runner/work/pyhf/pyhf/src/pyhf/tensor/manager.py", line 192, in set_backend
    events.trigger("tensorlib_changed")()
  File "/Users/runner/work/pyhf/pyhf/src/pyhf/events.py", line 70, in __call__
    func()(arg(), *args, **kwargs)
  File "/Users/runner/work/pyhf/pyhf/src/pyhf/tensor/common.py", line 33, in _precompute
    self.sorted_indices = tensorlib.astensor(self._sorted_indices, dtype='int')
AttributeError: 'NoneType' object has no attribute '_sorted_indices'
/Users/runner/work/pyhf/pyhf/src/pyhf/tensor/jax_backend.py:213: UnexpectedException
------------------------------ Captured log call -------------------------------
INFO     absl:xla_bridge.py:174 Remote TPU is not linked into jax; skipping remote TPU.
INFO     absl:xla_bridge.py:350 Unable to initialize backend 'tpu_driver': Could not initialize backend 'tpu_driver'
INFO     absl:xla_bridge.py:350 Unable to initialize backend 'cuda': module 'jaxlib.xla_extension' has no attribute 'GpuAllocatorConfig'
INFO     absl:xla_bridge.py:350 Unable to initialize backend 'rocm': module 'jaxlib.xla_extension' has no attribute 'GpuAllocatorConfig'
INFO     absl:xla_bridge.py:350 Unable to initialize backend 'tpu': module 'jaxlib.xla_extension' has no attribute 'get_tpu_client'
```


### pyhf Version

```console
PR #1274
```


### Code of Conduct

- [X] I agree to follow the Code of Conduct


Discussion:
@kratsg @lukasheinrich If I run our CI from scratch inside of a `python:3.10` Docker container I am unable to reproduce the error, which makes me think we're seeing a transient issue with GitHub Actions and not something real on our side.

~~Should we just go ahead and merge PR #1274 and hope for the best while opening up some GitHub Issues with the virtual environment team?~~ Though this isn't happening on `master` at f7e974f7c65524e5240c871154c00ba8e954f0bd, so there is something in the PR #1274 code that is causing this on the GitHub Action runners. :?
Expanding on https://github.com/scikit-hep/pyhf/pull/1274#discussion_r975974433, this is able to be reproduced locally if one comments out one of the docstrings of any of the 3 functions under the `pyhf.infer.intervals.upper_limits` API

```diff
> git diff
diff --git a/src/pyhf/infer/intervals/upper_limits.py b/src/pyhf/infer/intervals/upper_limits.py
index 5cc93c13..c609091b 100644
--- a/src/pyhf/infer/intervals/upper_limits.py
+++ b/src/pyhf/infer/intervals/upper_limits.py
@@ -33,22 +33,22 @@ def toms748_scan(
     Parameter of Interest (POI) using an automatic scan through
     POI-space, using the :func:`~scipy.optimize.toms748` algorithm.
 
-    Example:
-        >>> import numpy as np
-        >>> import pyhf
-        >>> pyhf.set_backend("numpy")
-        >>> model = pyhf.simplemodels.uncorrelated_background(
-        ...     signal=[12.0, 11.0], bkg=[50.0, 52.0], bkg_uncertainty=[3.0, 7.0]
-        ... )
-        >>> observations = [51, 48]
-        >>> data = pyhf.tensorlib.astensor(observations + model.config.auxdata)
-        >>> obs_limit, exp_limits = pyhf.infer.intervals.upper_limits.toms748_scan(
-        ...     data, model, 0., 5., rtol=0.01
-        ... )
-        >>> obs_limit
-        array(1.01156939)
-        >>> exp_limits
-        [array(0.5600747), array(0.75702605), array(1.06234693), array(1.50116923), array(2.05078912)]
+    # Example:
+    #     >>> import numpy as np
+    #     >>> import pyhf
+    #     >>> pyhf.set_backend("numpy")
+    #     >>> model = pyhf.simplemodels.uncorrelated_background(
+    #     ...     signal=[12.0, 11.0], bkg=[50.0, 52.0], bkg_uncertainty=[3.0, 7.0]
+    #     ... )
+    #     >>> observations = [51, 48]
+    #     >>> data = pyhf.tensorlib.astensor(observations + model.config.auxdata)
+    #     >>> obs_limit, exp_limits = pyhf.infer.intervals.upper_limits.toms748_scan(
+    #     ...     data, model, 0., 5., rtol=0.01
+    #     ... )
+    #     >>> obs_limit
+    #     array(1.01156939)
+    #     >>> exp_limits
+    #     [array(0.5600747), array(0.75702605), array(1.06234693), array(1.50116923), array(2.05078912)]
 
     Args:
         data (:obj:`tensor`): The observed data.
@@ -231,7 +231,6 @@ def upper_limit(
         array(1.01764089)
         >>> exp_limits
         [array(0.59576921), array(0.76169166), array(1.08504773), array(1.50170482), array(2.06654952)]
-        >>> pyhf.set_backend("jax")
 
     Args:
         data (:obj:`tensor`): The observed data.
```

and then run

```conosle
$ pytest \
          src/pyhf/infer/intervals/ \
          src/pyhf/tensor/jax_backend.py
================================================================================= test session starts =================================================================================
platform linux -- Python 3.10.6, pytest-7.1.3, pluggy-1.0.0
benchmark: 3.4.1 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000)
Matplotlib: 3.6.0
Freetype: 2.6.1
rootdir: /home/feickert/Code/GitHub/pyhf, configfile: pyproject.toml
plugins: socket-0.5.1, console-scripts-1.3.1, benchmark-3.4.1, cov-3.0.0, mock-3.8.2, mpl-0.16.1, requests-mock-1.10.0, anyio-3.6.1
collected 18 items                                                                                                                                                                    

src/pyhf/infer/intervals/upper_limits.py ..                                                                                                                                     [ 11%]
src/pyhf/tensor/jax_backend.py F...............                                                                                                                                 [100%]

====================================================================================== FAILURES =======================================================================================
_______________________________________________________________ [doctest] pyhf.tensor.jax_backend.jax_backend.astensor ________________________________________________________________
207 
208         Convert to a JAX ndarray.
209 
210         Example:
211 
212             >>> import pyhf
213             >>> pyhf.set_backend("jax")
UNEXPECTED EXCEPTION: AttributeError("'NoneType' object has no attribute '_sorted_indices'")
Traceback (most recent call last):
  File "/home/feickert/.pyenv/versions/3.10.6/lib/python3.10/doctest.py", line 1350, in __run
    exec(compile(example.source, filename, "single",
  File "<doctest pyhf.tensor.jax_backend.jax_backend.astensor[1]>", line 1, in <module>
  File "/home/feickert/Code/GitHub/pyhf/src/pyhf/events.py", line 161, in register_wrapper
    result = func(*args, **kwargs)
  File "/home/feickert/Code/GitHub/pyhf/src/pyhf/tensor/manager.py", line 192, in set_backend
    events.trigger("tensorlib_changed")()
  File "/home/feickert/Code/GitHub/pyhf/src/pyhf/events.py", line 70, in __call__
    func()(arg(), *args, **kwargs)
  File "/home/feickert/Code/GitHub/pyhf/src/pyhf/tensor/common.py", line 33, in _precompute
    self.sorted_indices = tensorlib.astensor(self._sorted_indices, dtype='int')
AttributeError: 'NoneType' object has no attribute '_sorted_indices'
/home/feickert/Code/GitHub/pyhf/src/pyhf/tensor/jax_backend.py:213: UnexpectedException
---------------------------------------------------------------------------------- Captured log call ----------------------------------------------------------------------------------
INFO     absl:xla_bridge.py:350 Unable to initialize backend 'tpu_driver': NOT_FOUND: Unable to find driver in registry given worker: 
INFO     absl:xla_bridge.py:350 Unable to initialize backend 'cuda': module 'jaxlib.xla_extension' has no attribute 'GpuAllocatorConfig'
INFO     absl:xla_bridge.py:350 Unable to initialize backend 'rocm': module 'jaxlib.xla_extension' has no attribute 'GpuAllocatorConfig'
INFO     absl:xla_bridge.py:350 Unable to initialize backend 'tpu': INVALID_ARGUMENT: TpuPlatform is not available.
WARNING  absl:xla_bridge.py:357 No GPU/TPU found, falling back to CPU. (Set TF_CPP_MIN_LOG_LEVEL=0 and rerun for more info.)

```

I think this might mean we have a bug in [`pyhf.tensor.manager`](https://github.com/scikit-hep/pyhf/blob/f7e974f7c65524e5240c871154c00ba8e954f0bd/src/pyhf/tensor/manager.py) or in how

https://github.com/scikit-hep/pyhf/blob/f7e974f7c65524e5240c871154c00ba8e954f0bd/tests/conftest.py#L49-L56

is used by doctest. Not sure, so input welcome.

Resolving patch:
diff --git a/src/conftest.py b/src/conftest.py
new file mode 120000
--- /dev/null
+++ b/src/conftest.py
@@ -0,0 +1 @@
+../tests/conftest.py
\ No newline at end of file


Test patch: