Repository: scikit-hep/pyhf

Issue:
Use of xfail in conftest.py hiding passing conditions
### Summary

In PR #1702 we adopted using `xfail_strict = true` for `pytest` to become aware of when tests that _should_ be failing aren't. With the release of `jax` `v0.2.26` (which includes a fix for https://github.com/google/jax/issues/8513 &mdash; c.f. PR #817)

https://github.com/scikit-hep/pyhf/blob/43c156702fd2a0392114c5b2468f0128ed16ea72/tests/test_tensor.py#L379-L396

_should_ be failing given `xfail_strict` as `jax` `v0.2.26` should be passing here. However, they don't fail.

This is because in `conftest.py` we use [`pytest.xfail`](https://docs.pytest.org/en/6.2.x/reference.html#pytest-xfail)

https://github.com/scikit-hep/pyhf/blob/43c156702fd2a0392114c5b2468f0128ed16ea72/tests/conftest.py#L101-L102

and [according to the xfail tutorial](https://docs.pytest.org/en/6.2.x/skipping.html#xfail)

> Note that no other code is executed after the `pytest.xfail()` call, differently from the marker

so the use of `pytest.xfail` in `conftest.py` is actually _skipping_ tests instead of running them and checking for failure (which is a surprise to me!). The [`pytest.xfail` docs](https://docs.pytest.org/en/6.2.x/reference.html#pytest-xfail) note that

> It is better to use the [`pytest.mark.xfail`](https://docs.pytest.org/en/6.2.x/reference.html#pytest-mark-xfail-ref) marker when possible to declare a test to be xfailed under certain conditions like known bugs or missing features.

Note also that

https://github.com/scikit-hep/pyhf/blob/43c156702fd2a0392114c5b2468f0128ed16ea72/pyproject.toml#L41-L42

should actually be under `[tool.pytest.ini_options]` as [noted in the `pytest` docs](https://docs.pytest.org/en/6.2.x/customize.html#pyproject-toml) (this was done wrong in PR #1702).

### OS / Environment

```console
All
```


### Steps to Reproduce

```diff
$ git diff
diff --git a/pyproject.toml b/pyproject.toml
index 9c81dc60..6616d586 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -38,11 +38,9 @@ ignore = [
     'AUTHORS',
 ]
 
-[tool.pytest]
-xfail_strict = true
-
 [tool.pytest.ini_options]
 minversion = "6.0"
+xfail_strict = true
 addopts = [
     "--ignore=setup.py",
     "--ignore=validation/",
diff --git a/tests/conftest.py b/tests/conftest.py
index 0e2537b7..e1c3f87e 100644
--- a/tests/conftest.py
+++ b/tests/conftest.py
@@ -99,7 +99,9 @@ def backend(request):
         )
 
     if fail_backend:
+        print(f"expect {func_name} to fail as specified")
         pytest.xfail(f"expect {func_name} to fail as specified")
+        print("this won't ever run")
 
     # actual execution here, after all checks is done
     pyhf.set_backend(*request.param)
diff --git a/tests/test_tensor.py b/tests/test_tensor.py
index 616b6ca5..3b8768e2 100644
--- a/tests/test_tensor.py
+++ b/tests/test_tensor.py
@@ -377,6 +377,7 @@ def test_boolean_mask(backend):
 
 
 @pytest.mark.fail_jax
+@pytest.mark.fail_numpy
 def test_percentile(backend):
     tb = pyhf.tensorlib
     a = tb.astensor([[10, 7, 4], [3, 2, 1]])
```

`test_percentile[numpy]` should pass, but use of `pytest.xfail` skips it:

```console
$ pytest -sx tests/test_tensor.py -k test_percentile[numpy]
=========================================================================================== test session starts ===========================================================================================
platform linux -- Python 3.9.6, pytest-6.2.5, py-1.10.0, pluggy-1.0.0
Matplotlib: 3.5.0
Freetype: 2.6.1
benchmark: 3.4.1 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000)
rootdir: /home/feickert/Code/GitHub/pyhf, configfile: pyproject.toml
plugins: mock-3.6.1, console-scripts-1.2.1, mpl-0.13, requests-mock-1.9.3, benchmark-3.4.1, cov-3.0.0, anyio-3.3.3
collected 222 items / 221 deselected / 1 selected                                                                                                                                                         

tests/test_tensor.py expect test_percentile[numpy] to fail as specified
x
```

### Expected Results

For tests marked with `xfail` to fail:

```diff
$ git diff
diff --git a/pyproject.toml b/pyproject.toml
index 9c81dc60..6616d586 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -38,11 +38,9 @@ ignore = [
     'AUTHORS',
 ]
 
-[tool.pytest]
-xfail_strict = true
-
 [tool.pytest.ini_options]
 minversion = "6.0"
+xfail_strict = true
 addopts = [
     "--ignore=setup.py",
     "--ignore=validation/",
diff --git a/tests/test_tensor.py b/tests/test_tensor.py
index 616b6ca5..421015f3 100644
--- a/tests/test_tensor.py
+++ b/tests/test_tensor.py
@@ -377,6 +377,7 @@ def test_boolean_mask(backend):
 
 
 @pytest.mark.fail_jax
+@pytest.mark.xfail()
 def test_percentile(backend):
     tb = pyhf.tensorlib
     a = tb.astensor([[10, 7, 4], [3, 2, 1]])
```

```console
$ pytest -sx tests/test_tensor.py -k test_percentile[numpy]
=========================================================================================== test session starts ===========================================================================================
platform linux -- Python 3.9.6, pytest-6.2.5, py-1.10.0, pluggy-1.0.0
Matplotlib: 3.5.0
Freetype: 2.6.1
benchmark: 3.4.1 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000)
rootdir: /home/feickert/Code/GitHub/pyhf, configfile: pyproject.toml
plugins: mock-3.6.1, console-scripts-1.2.1, mpl-0.13, requests-mock-1.9.3, benchmark-3.4.1, cov-3.0.0, anyio-3.3.3
collected 222 items / 221 deselected / 1 selected                                                                                                                                                         

tests/test_tensor.py F

================================================================================================ FAILURES =================================================================================================
_________________________________________________________________________________________ test_percentile[numpy] __________________________________________________________________________________________
[XPASS(strict)] 
```

### Actual Results

```console
$ pytest -sx tests/test_tensor.py -k test_percentile[numpy]
=========================================================================================== test session starts ===========================================================================================
platform linux -- Python 3.9.6, pytest-6.2.5, py-1.10.0, pluggy-1.0.0
Matplotlib: 3.5.0
Freetype: 2.6.1
benchmark: 3.4.1 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000)
rootdir: /home/feickert/Code/GitHub/pyhf, configfile: pyproject.toml
plugins: mock-3.6.1, console-scripts-1.2.1, mpl-0.13, requests-mock-1.9.3, benchmark-3.4.1, cov-3.0.0, anyio-3.3.3
collected 222 items / 221 deselected / 1 selected                                                                                                                                                         

tests/test_tensor.py expect test_percentile[numpy] to fail as specified
x
```


### pyhf Version

`master` at commit 43c156702fd2a0392114c5b2468f0128ed16ea72


### Code of Conduct

- [X] I agree to follow the Code of Conduct


Discussion:

Resolving patch:
diff --git a/src/pyhf/tensor/jax_backend.py b/src/pyhf/tensor/jax_backend.py
--- a/src/pyhf/tensor/jax_backend.py
+++ b/src/pyhf/tensor/jax_backend.py
@@ -287,7 +287,7 @@ def percentile(self, tensor_in, q, axis=None, interpolation="linear"):
             >>> import jax.numpy as jnp
             >>> pyhf.set_backend("jax")
             >>> a = pyhf.tensorlib.astensor([[10, 7, 4], [3, 2, 1]])
-            >>> pyhf.tensorlib.percentile(a, jnp.float64(50))
+            >>> pyhf.tensorlib.percentile(a, 50)
             DeviceArray(3.5, dtype=float64)
             >>> pyhf.tensorlib.percentile(a, 50, axis=1)
             DeviceArray([7., 2.], dtype=float64)
@@ -314,8 +314,6 @@ def percentile(self, tensor_in, q, axis=None, interpolation="linear"):
             JAX ndarray: The value of the :math:`q`-th percentile of the tensor along the specified axis.
 
         """
-        # TODO: Monitor future JAX releases for changes to percentile dtype promotion
-        # c.f. https://github.com/google/jax/issues/8513
         return jnp.percentile(tensor_in, q, axis=axis, interpolation=interpolation)
 
     def stack(self, sequence, axis=0):


Test patch:
diff --git a/tests/conftest.py b/tests/conftest.py
--- a/tests/conftest.py
+++ b/tests/conftest.py
@@ -99,7 +99,13 @@ def backend(request):
         )
 
     if fail_backend:
-        pytest.xfail(f"expect {func_name} to fail as specified")
+        # Mark the test as xfail to actually run it and ensure that it does
+        # fail. pytest.mark.xfail checks for failure, while pytest.xfail
+        # assumes failure and skips running the test.
+        # c.f. https://docs.pytest.org/en/6.2.x/skipping.html#xfail
+        request.node.add_marker(
+            pytest.mark.xfail(reason=f"expect {func_name} to fail as specified")
+        )
 
     # actual execution here, after all checks is done
     pyhf.set_backend(*request.param)
diff --git a/tests/test_tensor.py b/tests/test_tensor.py
--- a/tests/test_tensor.py
+++ b/tests/test_tensor.py
@@ -376,7 +376,6 @@ def test_boolean_mask(backend):
     )
 
 
-@pytest.mark.fail_jax
 def test_percentile(backend):
     tb = pyhf.tensorlib
     a = tb.astensor([[10, 7, 4], [3, 2, 1]])
@@ -392,7 +391,6 @@ def test_percentile(backend):
 # c.f. https://github.com/scikit-hep/pyhf/issues/1693
 @pytest.mark.fail_pytorch
 @pytest.mark.fail_pytorch64
-@pytest.mark.fail_jax
 def test_percentile_interpolation(backend):
     tb = pyhf.tensorlib
     a = tb.astensor([[10, 7, 4], [3, 2, 1]])
@@ -404,33 +402,6 @@ def test_percentile_interpolation(backend):
     assert tb.tolist(tb.percentile(a, 50, interpolation="higher")) == 4.0
 
 
-@pytest.mark.only_jax
-def test_percentile_jax(backend):
-    tb = pyhf.tensorlib
-    a = tb.astensor([[10, 7, 4], [3, 2, 1]])
-    assert tb.tolist(tb.percentile(a, 0)) == 1
-
-    # TODO: Monitor future JAX releases for changes to percentile dtype promotion
-    # c.f. https://github.com/scikit-hep/pyhf/issues/1693
-    assert tb.tolist(tb.percentile(a, np.float64(50))) == 3.5
-    assert tb.tolist(tb.percentile(a, np.float64(100))) == 10
-    assert tb.tolist(tb.percentile(a, 50, axis=1)) == [7.0, 2.0]
-
-
-@pytest.mark.only_jax
-def test_percentile_interpolation_jax(backend):
-    tb = pyhf.tensorlib
-    a = tb.astensor([[10, 7, 4], [3, 2, 1]])
-
-    # TODO: Monitor future JAX releases for changes to percentile dtype promotion
-    # c.f. https://github.com/scikit-hep/pyhf/issues/1693
-    assert tb.tolist(tb.percentile(a, np.float64(50), interpolation="linear")) == 3.5
-    assert tb.tolist(tb.percentile(a, 50, interpolation="nearest")) == 3.0
-    assert tb.tolist(tb.percentile(a, 50, interpolation="lower")) == 3.0
-    assert tb.tolist(tb.percentile(a, 50, interpolation="midpoint")) == 3.5
-    assert tb.tolist(tb.percentile(a, 50, interpolation="higher")) == 4.0
-
-
 def test_tensor_tile(backend):
     a = [[1], [2], [3]]
     tb = pyhf.tensorlib
@@ -465,7 +436,6 @@ def test_1D_gather(backend):
     ) == [[5, 1], [4, 3]]
 
 
-@pytest.mark.fail_pytorch
 def test_ND_gather(backend):
     tb = pyhf.tensorlib
     assert tb.tolist(