Repository: scikit-hep/pyhf

Issue:
weakrefs to dead objects occuring when changing backends in pyhf benchmark
### Summary

I try to perform a benchmark of `pyhf` using `pytest-benchmark` quite similarly to the benchmark in the `tests/benchmarks` directory.
A short example of such a simple benchmark is given below. To reproduce this bug, the python code needs to be saved in a file of the format `test_<name>.py` and executed via `pytest test_<name>.py`.
The bug occurs only sometimes when the backend is changed between different benchmarking cases. Since the occurence of the bug includes some amount of randomness, it may happen that it doesn't occur on the first try but that the benchmark must be executed multiple times. The full benchmark takes around 1 min on my machine.

The suspected origin of this bug is that every time the backend is changed, an event called `tensorlib_changed` is triggered that in turn leads to the execution of some `_precompute()` functions on different objects (e.g. a `TensorViewer` object as in the error message). The problem occurs, when the referenced object no longer exists, as all references to it have been removed. The reference used to call the function does not change this as it is a weakref.

A proposed solution can be found in PR #2310. 

### OS / Environment

```console
PRETTY_NAME="Ubuntu 22.04.3 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.3 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy
```


### Steps to Reproduce

<!--- Paste your minimal failing Python example code between the quotes below -->
```python (paste below)
# content of test_benchmark.py
import pytest
import pyhf

@pytest.fixture(
    scope='function',
    params=[
        (pyhf.tensor.numpy_backend(), None),
        (pyhf.tensor.pytorch_backend(), None),
        (pyhf.tensor.pytorch_backend(precision='64b'), None),
        (pyhf.tensor.tensorflow_backend(), None),
        (pyhf.tensor.jax_backend(), None),
        (
            pyhf.tensor.numpy_backend(poisson_from_normal=True),
            pyhf.optimize.minuit_optimizer(),
        ),
    ],
    ids=['numpy', 'pytorch', 'pytorch64',
         'tensorflow',
         'jax', 'numpy_minuit'],
)
def backend(request):
    # get the ids of all the backends
    param_ids = request._fixturedef.ids
    # the backend we're using: numpy, tensorflow, etc...
    param_id = param_ids[request.param_index]
    # name of function being called (with params), the original name is .originalname
    func_name = request._pyfuncitem.name

    pyhf.set_backend(*request.param)

    yield request.param

def hypotest(data, pdf):
    return pyhf.infer.hypotest(1.0, data, pdf, test_stat="qtilde", return_expected=True)

bins = [1, 2, 4, 8, 16, 32]
bin_ids = [f'{n_bins}_bins' for n_bins in bins]

@pytest.mark.parametrize('n_bins', bins, ids=bin_ids)
def test_hypotest(benchmark, backend, n_bins):
    model = pyhf.simplemodels.uncorrelated_background(signal=[12.0]*n_bins, bkg=[50.0]*n_bins, bkg_uncertainty=[5.0]*n_bins)
    data = [51.0]*n_bins + model.config.auxdata

    assert benchmark(hypotest, data, model)
```

<!--- ...or if you have a failing CLI command paste it between the quotes below -->
```console (paste below)
pytest test_benchmark.py
```


### File Upload (optional)

_No response_

### Expected Results

The expected behavior is to output the benchmarking results for all considered cases as it can be observed when executing `pytest` in `pyhf/tests/benchmarks/`.
This output should not show any "test failures" as no normal tests are performed but only functions that run without an error, when called outside of the benchmark.

### Actual Results

```console
_________________________ ERROR at setup of test_hypotest[jax-1_bins] _________________________

request = <SubRequest 'backend' for <Function test_hypotest[jax-1_bins]>>

    @pytest.fixture(
        scope='function',
        params=[
            (pyhf.tensor.numpy_backend(), None),
            (pyhf.tensor.pytorch_backend(), None),
            (pyhf.tensor.pytorch_backend(precision='64b'), None),
            (pyhf.tensor.tensorflow_backend(), None),
            (pyhf.tensor.jax_backend(), None),
            (
                pyhf.tensor.numpy_backend(poisson_from_normal=True),
                pyhf.optimize.minuit_optimizer(),
            ),
        ],
        ids=['numpy', 'pytorch', 'pytorch64',
             'tensorflow',
             'jax', 'numpy_minuit'],
    )
    def backend(request):
        # get the ids of all the backends
        param_ids = request._fixturedef.ids
        # the backend we're using: numpy, tensorflow, etc...
        param_id = param_ids[request.param_index]
        # name of function being called (with params), the original name is .originalname
        func_name = request._pyfuncitem.name
    
>       pyhf.set_backend(*request.param)

test_hypo_pyhf.py:29: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
../../pyhfDev/pyhf/src/pyhf/events.py:161: in register_wrapper
    result = func(*args, **kwargs)
../../pyhfDev/pyhf/src/pyhf/tensor/manager.py:193: in set_backend
    events.trigger("tensorlib_changed")()
../../pyhfDev/pyhf/src/pyhf/events.py:70: in __call__
    func()(arg(), *args, **kwargs)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = None

    def _precompute(self):
        tensorlib, _ = get_backend()
>       self.sorted_indices = tensorlib.astensor(self._sorted_indices, dtype='int')
E       AttributeError: 'NoneType' object has no attribute '_sorted_indices'

../../pyhfDev/pyhf/src/pyhf/tensor/common.py:33: AttributeError
```


### pyhf Version

```console
pyhf, version 0.7.1.dev116
```


### Code of Conduct

- [X] I agree to follow the Code of Conduct


Discussion:

Resolving patch:
diff --git a/src/pyhf/events.py b/src/pyhf/events.py
--- a/src/pyhf/events.py
+++ b/src/pyhf/events.py
@@ -64,12 +64,20 @@ def _flush(self):
         self._callbacks = _callbacks
 
     def __call__(self, *args, **kwargs):
-        for func, arg in self.callbacks:
+        for func, arg in self._callbacks:
             # weakref: needs to be de-ref'd first before calling
             if arg is not None:
-                func()(arg(), *args, **kwargs)
+                arg_ref = arg()
+                if arg_ref is not None:
+                    func()(arg_ref, *args, **kwargs)
             else:
                 func()(*args, **kwargs)
+        # Flush after calling all the callbacks, not before, as callbacks in the
+        # beginning of the iteration might cause new dead arg weakrefs in
+        # callbacks that are iterated over later.
+        # Checking for dead weakrefs in each iteration and flushing at the end
+        # avoids redundant dead weakref checking in subsequent calls.
+        self._flush()
 
     def __iter__(self):
         return iter(self.callbacks)


Test patch: