Repository: scikit-hep/pyhf

Issue:
Auto scan for hypothesis test inversion (Implemented)
# Description

An auto scan feature for hypothesis test inversion can provide faster and more precise upper limits. One good way I've found to do this is to use the [TOMS748](https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.toms748.html#scipy.optimize.toms748) root finding algorithm.

### Describe the solution you'd like
An `upperlimit_auto` (or some other appropriate name) function that runs an auto scan instead of a linear scan.

### Implementation
An implementation is below. I can make a PR including this in `pyhf.infer.intevals` if this issue has support.

```python
from scipy.optimize import toms748 as _toms748

def upperlimit_auto(data, model, low, high, level=0.05, atol=2e-12, rtol=1e-15):
    """
    Calculate an upper limit interval ``(0, poi_up)`` for a single
    Parameter of Interest (POI) using an automatic scan through
    POI-space, using the TOMS748 algorithm.
    
    ..., mostly copied from upperlimit docstring.
    """

    def f_all(mu):
        return hypotest(mu, data, model, test_stat="qtilde", return_expected_set=True)

    def f(mu, limit=0):
        # Use integers for limit so we don't need a string comparison
        if limit == 0:
            # Obs
            return f_all(mu)[0] - level
        else:
            # Exp
            # (These are in the order -2, -1, 0, 1, 2 sigma)
            return f_all(mu)[1][limit - 1] - level

    tb, _ = get_backend()
    obs = tb.astensor(_toms748(f, low, high, args=(0), k=2, xtol=atol, rtol=rtol))
    exp = [
        tb.astensor(_toms748(f, low, high, args=(i), k=2, xtol=atol, rtol=rtol))
        for i in range(1, 6)
    ]
    return obs, exp
```


Discussion:
There is a brent-bracketing implementation by @alexander-held in `cabinetry` (https://github.com/alexander-held/cabinetry/blob/c85f404bc0bdedd3bb2629c826de385b694d99fc/src/cabinetry/fit.py#L589) that does a non-linear limit finding. If this is something people want to see upstreamed, we could.
It would be ince to find a way to be able to inject algorithms that can perform scans without taking them on fully (maybe ismilar to the work on toy calculator API)..  scikit optimize has the ask/tell API

https://scikit-optimize.github.io/stable/auto_examples/ask-and-tell.html

probably would fit into `pyhf.infer.intervals`
I think for future-proofing it would be good if this returns "intervals" and  not only try to do a root-finding.. i.e. the implicit assumption is often that the itnerval is [0,root] annd this should be made explicit
I think it would be really helpful to have an auto scan in pyhf itself for those who don't use Cabinetry, particularly since a linear scan _is_ available.

> It would be ince to find a way to be able to inject algorithms that can perform scans without taking them on fully

I can easily extend the implementation I posted here to allow the user to change the root-finding function used (assuming it uses the bracketing API shared by the `scipy.optimize` bracketing root-finders). That said, I think TOMS748 is usually the optimal choice (and [known to be better than Brent's method](https://www.boost.org/doc/libs/1_75_0/libs/math/doc/html/math_toolkit/roots_noderiv/brent.html)), so I do think it should be the default.

> I think for future-proofing it would be good if this returns "intervals" and not only try to do a root-finding

The `upperlimit_auto` function could certainly return `(0, upper_lim)` for each of the observed and expected limits. Above, I just copied the format that the linear scan returns, which is also why I wrapped each limit in a tensor.
Hi, providing this functionality through `pyhf.infer` sounds like a good idea to me. I have a few ideas and suggestions based on my experience with the `cabinetry` implementation:
- Caching is useful, the above algorithm will repeat fits for upper/lower bracket mu values for every root finding (obs + 5x for exp, resulting in at least 10 repeated `infer.hypotest` calls).
- The starting bracket for subsequent limits (e.g. the exp limits after having calculated obs limits) can be optimized given information about CLs(mu) obtained from the previous steps.
- In a test case, the algorithm repeatedly queries the same mu for a single root finding step, example below. I am not sure what causes this, `brentq` does not show the same behavior (thank you for the pointer to `toms748`, will try that out in more detail!).
```
testing mu=0
testing mu=10
testing mu=9.497575427630334
testing mu=6.715660956538139
testing mu=6.715660956538139
testing mu=6.04083915699075
testing mu=3.020419578495375
testing mu=2.633299000060376
testing mu=2.633299000060376
testing mu=2.43394097830554
testing mu=1.21697048915277
testing mu=1.7419775663353678
testing mu=2.0585075562095017
testing mu=1.9611870482306077
```

In the longer term, I believe that an implementation which uses knowledge about the functional form of CLs(mu) under asymptotic assumptions can likely be significantly more efficient than pure root finding. An implementation of such a function-aware limit finding algorithm seems very useful when using `pyhf` to calculate limits with complex workspaces where each fit takes a non-negligible amount of time (there is a script in used internally in ATLAS that is very efficient in my experience).
> Caching is useful, the above algorithm will repeat fits for upper/lower bracket mu values for every root finding (obs + 5x for exp, resulting in at least 10 repeated infer.hypotest calls).

I figured that would be the case. My test implementation actually has an `@lru_cache` decorator on `f_all` but I removed it for the snippet above to remove an import. Given your third point, it does seem to be useful to add that back in.

> The starting bracket for subsequent limits (e.g. the exp limits after having calculated obs limits) can be optimized given information about CLs(mu) obtained from the previous steps.

Agreed. This isn't possible using `lru_cache`, but having tried with a manually implemented cache, it does seem to improve things.

> In a test case, the algorithm repeatedly queries the same mu for a single root finding step

I see this too. I think it's a feature of the root finding algorithm. With memoization enabled it doesn't have much impact though.

> In the longer term, I believe that an implementation which uses knowledge about the functional form of CLs(mu) under asymptotic assumptions can likely be significantly more efficient than pure root finding.

This is true, but I don't think a root finding auto scan should be left out because we're waiting for a more direct / analytical implementation of limit setting. The root finding technique is also more general and can be used with a toy-based hypothesis test implementation.

The ATLAS script you refer to does use root finding (I think it uses the secant method), it's efficiency stems from a clever choice of starting value which means you don't need many iterations of the root finding algorithm.
> I don't think a root finding auto scan should be left out because we're waiting for a more direct / analytical implementation of limit setting

I agree and would view the different implementations as complementary.

> The ATLAS script you refer to does use root finding (I think it uses the secant method), it's efficiency stems from a clever choice of starting value which means you don't need many iterations of the root finding algorithm.

Thanks for the details, I will have to take a closer look at this implementation.

Resolving patch:
diff --git a/src/pyhf/infer/intervals.py b/src/pyhf/infer/intervals.py
deleted file mode 100644
--- a/src/pyhf/infer/intervals.py
+++ /dev/null
@@ -1,75 +0,0 @@
-"""Interval estimation"""
-from pyhf.infer import hypotest
-from pyhf import get_backend
-import numpy as np
-
-__all__ = ["upperlimit"]
-
-
-def __dir__():
-    return __all__
-
-
-def _interp(x, xp, fp):
-    tb, _ = get_backend()
-    return tb.astensor(np.interp(x, xp.tolist(), fp.tolist()))
-
-
-def upperlimit(data, model, scan, level=0.05, return_results=False, **hypotest_kwargs):
-    """
-    Calculate an upper limit interval ``(0, poi_up)`` for a single
-    Parameter of Interest (POI) using a fixed scan through POI-space.
-
-    Example:
-        >>> import numpy as np
-        >>> import pyhf
-        >>> pyhf.set_backend("numpy")
-        >>> model = pyhf.simplemodels.uncorrelated_background(
-        ...     signal=[12.0, 11.0], bkg=[50.0, 52.0], bkg_uncertainty=[3.0, 7.0]
-        ... )
-        >>> observations = [51, 48]
-        >>> data = pyhf.tensorlib.astensor(observations + model.config.auxdata)
-        >>> scan = np.linspace(0, 5, 21)
-        >>> obs_limit, exp_limits, (scan, results) = pyhf.infer.intervals.upperlimit(
-        ...     data, model, scan, return_results=True
-        ... )
-        >>> obs_limit
-        array(1.01764089)
-        >>> exp_limits
-        [array(0.59576921), array(0.76169166), array(1.08504773), array(1.50170482), array(2.06654952)]
-
-    Args:
-        data (:obj:`tensor`): The observed data.
-        model (~pyhf.pdf.Model): The statistical model adhering to the schema ``model.json``.
-        scan (:obj:`iterable`): Iterable of POI values.
-        level (:obj:`float`): The threshold value to evaluate the interpolated results at.
-        return_results (:obj:`bool`): Whether to return the per-point results.
-        hypotest_kwargs (:obj:`string`): Kwargs for the calls to
-         :class:`~pyhf.infer.hypotest` to configure the fits.
-
-    Returns:
-        Tuple of Tensors:
-
-            - Tensor: The observed upper limit on the POI.
-            - Tensor: The expected upper limits on the POI.
-            - Tuple of Tensors: The given ``scan`` along with the
-              :class:`~pyhf.infer.hypotest` results at each test POI.
-              Only returned when ``return_results`` is ``True``.
-    """
-    tb, _ = get_backend()
-    results = [
-        hypotest(mu, data, model, return_expected_set=True, **hypotest_kwargs)
-        for mu in scan
-    ]
-    obs = tb.astensor([[r[0]] for r in results])
-    exp = tb.astensor([[r[1][idx] for idx in range(5)] for r in results])
-
-    result_arrary = tb.concatenate([obs, exp], axis=1).T
-
-    # observed limit and the (0, +-1, +-2)sigma expected limits
-    limits = [_interp(level, result_arrary[idx][::-1], scan[::-1]) for idx in range(6)]
-    obs_limit, exp_limits = limits[0], limits[1:]
-
-    if return_results:
-        return obs_limit, exp_limits, (scan, results)
-    return obs_limit, exp_limits
diff --git a/src/pyhf/infer/intervals/__init__.py b/src/pyhf/infer/intervals/__init__.py
new file mode 100644
--- /dev/null
+++ b/src/pyhf/infer/intervals/__init__.py
@@ -0,0 +1,30 @@
+"""Interval estimation"""
+import pyhf.infer.intervals.upper_limits
+
+__all__ = ["upper_limits.upper_limit"]
+
+
+def __dir__():
+    return __all__
+
+
+def upperlimit(
+    data, model, scan=None, level=0.05, return_results=False, **hypotest_kwargs
+):
+    """
+    .. deprecated:: 0.7.0
+       Use :func:`~pyhf.infer.intervals.upper_limits.upper_limit` instead.
+    .. warning:: :func:`~pyhf.infer.intervals.upperlimit` will be removed in
+     ``pyhf`` ``v0.9.0``.
+    """
+    from pyhf.exceptions import _deprecated_api_warning
+
+    _deprecated_api_warning(
+        "pyhf.infer.intervals.upperlimit",
+        "pyhf.infer.intervals.upper_limits.upper_limit",
+        "0.7.0",
+        "0.9.0",
+    )
+    return pyhf.infer.intervals.upper_limits.upper_limit(
+        data, model, scan, level, return_results, **hypotest_kwargs
+    )
diff --git a/src/pyhf/infer/intervals/upper_limits.py b/src/pyhf/infer/intervals/upper_limits.py
new file mode 100644
--- /dev/null
+++ b/src/pyhf/infer/intervals/upper_limits.py
@@ -0,0 +1,269 @@
+"""Interval estimation"""
+import numpy as np
+from scipy.optimize import toms748
+
+from pyhf import get_backend
+from pyhf.infer import hypotest
+
+__all__ = ["upper_limit", "linear_grid_scan", "toms748_scan"]
+
+
+def __dir__():
+    return __all__
+
+
+def _interp(x, xp, fp):
+    tb, _ = get_backend()
+    return tb.astensor(np.interp(x, xp.tolist(), fp.tolist()))
+
+
+def toms748_scan(
+    data,
+    model,
+    bounds_low,
+    bounds_up,
+    level=0.05,
+    atol=2e-12,
+    rtol=1e-4,
+    from_upper_limit_fn=False,
+    **hypotest_kwargs,
+):
+    """
+    Calculate an upper limit interval ``(0, poi_up)`` for a single
+    Parameter of Interest (POI) using an automatic scan through
+    POI-space, using the :func:`~scipy.optimize.toms748` algorithm.
+
+    Example:
+        >>> import numpy as np
+        >>> import pyhf
+        >>> pyhf.set_backend("numpy")
+        >>> model = pyhf.simplemodels.uncorrelated_background(
+        ...     signal=[12.0, 11.0], bkg=[50.0, 52.0], bkg_uncertainty=[3.0, 7.0]
+        ... )
+        >>> observations = [51, 48]
+        >>> data = pyhf.tensorlib.astensor(observations + model.config.auxdata)
+        >>> obs_limit, exp_limits = pyhf.infer.intervals.upper_limits.toms748_scan(
+        ...     data, model, 0., 5., rtol=0.01
+        ... )
+        >>> obs_limit
+        array(1.01156939)
+        >>> exp_limits
+        [array(0.5600747), array(0.75702605), array(1.06234693), array(1.50116923), array(2.05078912)]
+
+    Args:
+        data (:obj:`tensor`): The observed data.
+        model (~pyhf.pdf.Model): The statistical model adhering to the schema ``model.json``.
+        bounds_low (:obj:`float`): Lower boundary of search interval.
+        bounds_up (:obj:`float`): Upper boundary of search interval.
+        level (:obj:`float`): The threshold value to evaluate the interpolated results at.
+                              Defaults to ``0.05``.
+        atol (:obj:`float`): Absolute tolerance.
+                             The iteration will end when the result is within absolute
+                             *or* relative tolerance of the true limit.
+        rtol (:obj:`float`): Relative tolerance.
+                             For optimal performance this argument should be set
+                             to the highest acceptable relative tolerance.
+        hypotest_kwargs (:obj:`string`): Kwargs for the calls to
+         :class:`~pyhf.infer.hypotest` to configure the fits.
+
+    Returns:
+        Tuple of Tensors:
+
+            - Tensor: The observed upper limit on the POI.
+            - Tensor: The expected upper limits on the POI.
+
+    .. versionadded:: 0.7.0
+    """
+    cache = {}
+
+    def f_cached(poi):
+        if poi not in cache:
+            cache[poi] = hypotest(
+                poi,
+                data,
+                model,
+                return_expected_set=True,
+                **hypotest_kwargs,
+            )
+        return cache[poi]
+
+    def f(poi, level, limit=0):
+        # Use integers for limit so we don't need a string comparison
+        # limit == 0: Observed
+        # else: expected
+        return (
+            f_cached(poi)[0] - level
+            if limit == 0
+            else f_cached(poi)[1][limit - 1] - level
+        )
+
+    def best_bracket(limit):
+        # return best bracket
+        ks = np.asarray(list(cache.keys()))
+        vals = np.asarray(
+            [
+                value[0] - level if limit == 0 else value[1][limit - 1] - level
+                for value in cache.values()
+            ]
+        )
+        pos = vals >= 0
+        neg = vals < 0
+        lower = ks[pos][np.argmin(vals[pos])]
+        upper = ks[neg][np.argmax(vals[neg])]
+        return (lower, upper)
+
+    # extend bounds_low and bounds_up if they don't bracket CLs level
+    lower_results = f_cached(bounds_low)
+    # {lower,upper}_results[0] is an array and {lower,upper}_results[1] is a
+    # list of arrays so need to turn {lower,upper}_results[0] into list to
+    # concatenate them
+    while np.any(np.asarray([lower_results[0]] + lower_results[1]) < level):
+        bounds_low /= 2
+        lower_results = f_cached(bounds_low)
+    upper_results = f_cached(bounds_up)
+    while np.any(np.asarray([upper_results[0]] + upper_results[1]) > level):
+        bounds_up *= 2
+        upper_results = f_cached(bounds_up)
+
+    tb, _ = get_backend()
+    obs = tb.astensor(
+        toms748(f, bounds_low, bounds_up, args=(level, 0), k=2, xtol=atol, rtol=rtol)
+    )
+    exp = [
+        tb.astensor(
+            toms748(f, *best_bracket(idx), args=(level, idx), k=2, xtol=atol, rtol=rtol)
+        )
+        for idx in range(1, 6)
+    ]
+    if from_upper_limit_fn:
+        return obs, exp, (list(cache.keys()), list(cache.values()))
+    return obs, exp
+
+
+def linear_grid_scan(
+    data, model, scan, level=0.05, return_results=False, **hypotest_kwargs
+):
+    """
+    Calculate an upper limit interval ``(0, poi_up)`` for a single
+    Parameter of Interest (POI) using a linear scan through POI-space.
+
+    Example:
+        >>> import numpy as np
+        >>> import pyhf
+        >>> pyhf.set_backend("numpy")
+        >>> model = pyhf.simplemodels.uncorrelated_background(
+        ...     signal=[12.0, 11.0], bkg=[50.0, 52.0], bkg_uncertainty=[3.0, 7.0]
+        ... )
+        >>> observations = [51, 48]
+        >>> data = pyhf.tensorlib.astensor(observations + model.config.auxdata)
+        >>> scan = np.linspace(0, 5, 21)
+        >>> obs_limit, exp_limits, (scan, results) = pyhf.infer.intervals.upper_limits.upper_limit(
+        ...     data, model, scan, return_results=True
+        ... )
+        >>> obs_limit
+        array(1.01764089)
+        >>> exp_limits
+        [array(0.59576921), array(0.76169166), array(1.08504773), array(1.50170482), array(2.06654952)]
+
+    Args:
+        data (:obj:`tensor`): The observed data.
+        model (~pyhf.pdf.Model): The statistical model adhering to the schema ``model.json``.
+        scan (:obj:`iterable`): Iterable of POI values.
+        level (:obj:`float`): The threshold value to evaluate the interpolated results at.
+        return_results (:obj:`bool`): Whether to return the per-point results.
+        hypotest_kwargs (:obj:`string`): Kwargs for the calls to
+         :class:`~pyhf.infer.hypotest` to configure the fits.
+
+    Returns:
+        Tuple of Tensors:
+
+            - Tensor: The observed upper limit on the POI.
+            - Tensor: The expected upper limits on the POI.
+            - Tuple of Tensors: The given ``scan`` along with the
+              :class:`~pyhf.infer.hypotest` results at each test POI.
+              Only returned when ``return_results`` is ``True``.
+
+    .. versionadded:: 0.7.0
+    """
+    tb, _ = get_backend()
+    results = [
+        hypotest(mu, data, model, return_expected_set=True, **hypotest_kwargs)
+        for mu in scan
+    ]
+    obs = tb.astensor([[r[0]] for r in results])
+    exp = tb.astensor([[r[1][idx] for idx in range(5)] for r in results])
+
+    result_array = tb.concatenate([obs, exp], axis=1).T
+
+    # observed limit and the (0, +-1, +-2)sigma expected limits
+    limits = [_interp(level, result_array[idx][::-1], scan[::-1]) for idx in range(6)]
+    obs_limit, exp_limits = limits[0], limits[1:]
+
+    if return_results:
+        return obs_limit, exp_limits, (scan, results)
+    return obs_limit, exp_limits
+
+
+def upper_limit(
+    data, model, scan=None, level=0.05, return_results=False, **hypotest_kwargs
+):
+    """
+    Calculate an upper limit interval ``(0, poi_up)`` for a single Parameter of
+    Interest (POI) using root-finding or a linear scan through POI-space.
+
+    Example:
+        >>> import numpy as np
+        >>> import pyhf
+        >>> pyhf.set_backend("numpy")
+        >>> model = pyhf.simplemodels.uncorrelated_background(
+        ...     signal=[12.0, 11.0], bkg=[50.0, 52.0], bkg_uncertainty=[3.0, 7.0]
+        ... )
+        >>> observations = [51, 48]
+        >>> data = pyhf.tensorlib.astensor(observations + model.config.auxdata)
+        >>> scan = np.linspace(0, 5, 21)
+        >>> obs_limit, exp_limits, (scan, results) = pyhf.infer.intervals.upper_limits.upper_limit(
+        ...     data, model, scan, return_results=True
+        ... )
+        >>> obs_limit
+        array(1.01764089)
+        >>> exp_limits
+        [array(0.59576921), array(0.76169166), array(1.08504773), array(1.50170482), array(2.06654952)]
+
+    Args:
+        data (:obj:`tensor`): The observed data.
+        model (~pyhf.pdf.Model): The statistical model adhering to the schema ``model.json``.
+        scan (:obj:`iterable` or ``None``): Iterable of POI values or ``None`` to use
+         :class:`~pyhf.infer.intervals.upper_limits.toms748_scan`.
+        level (:obj:`float`): The threshold value to evaluate the interpolated results at.
+        return_results (:obj:`bool`): Whether to return the per-point results.
+
+    Returns:
+        Tuple of Tensors:
+
+            - Tensor: The observed upper limit on the POI.
+            - Tensor: The expected upper limits on the POI.
+            - Tuple of Tensors: The given ``scan`` along with the
+              :class:`~pyhf.infer.hypotest` results at each test POI.
+              Only returned when ``return_results`` is ``True``.
+
+    .. versionadded:: 0.7.0
+    """
+    if scan is not None:
+        return linear_grid_scan(
+            data, model, scan, level, return_results, **hypotest_kwargs
+        )
+    # else:
+    bounds = model.config.suggested_bounds()[
+        model.config.par_slice(model.config.poi_name).start
+    ]
+    obs_limit, exp_limit, results = toms748_scan(
+        data,
+        model,
+        bounds[0],
+        bounds[1],
+        from_upper_limit_fn=True,
+        **hypotest_kwargs,
+    )
+    if return_results:
+        return obs_limit, exp_limit, results
+    return obs_limit, exp_limit


Test patch:
diff --git a/tests/constraints.txt b/tests/constraints.txt
--- a/tests/constraints.txt
+++ b/tests/constraints.txt
@@ -1,5 +1,5 @@
 # core
-scipy==1.1.0
+scipy==1.2.0  # c.f. PR #1274
 click==8.0.0  # c.f. PR #1958, #1909
 tqdm==4.56.0
 jsonschema==4.15.0  # c.f. PR #1979
diff --git a/tests/test_infer.py b/tests/test_infer.py
--- a/tests/test_infer.py
+++ b/tests/test_infer.py
@@ -1,8 +1,11 @@
-import pytest
-import pyhf
+import warnings
+
 import numpy as np
+import pytest
 import scipy.stats
 
+import pyhf
+
 
 @pytest.fixture(scope="function")
 def hypotest_args():
@@ -20,12 +23,91 @@ def check_uniform_type(in_list):
     )
 
 
-def test_upperlimit(tmpdir, hypotest_args):
+def test_toms748_scan(tmpdir, hypotest_args):
+    """
+    Test the upper limit toms748 scan returns the correct structure and values
+    """
+    _, data, model = hypotest_args
+    results = pyhf.infer.intervals.upper_limits.toms748_scan(
+        data, model, 0, 5, rtol=1e-8
+    )
+    assert len(results) == 2
+    observed_limit, expected_limits = results
+    observed_cls = pyhf.infer.hypotest(
+        observed_limit,
+        data,
+        model,
+        model.config.suggested_init(),
+        model.config.suggested_bounds(),
+    )
+    expected_cls = np.array(
+        [
+            pyhf.infer.hypotest(
+                expected_limits[i],
+                data,
+                model,
+                model.config.suggested_init(),
+                model.config.suggested_bounds(),
+                return_expected_set=True,
+            )[1][i]
+            for i in range(5)
+        ]
+    )
+    assert observed_cls == pytest.approx(0.05)
+    assert expected_cls == pytest.approx(0.05)
+
+
+def test_toms748_scan_bounds_extension(hypotest_args):
+    """
+    Test the upper limit toms748 scan bounds can correctly extend to bracket the CLs level
+    """
+    _, data, model = hypotest_args
+    results_default = pyhf.infer.intervals.upper_limits.toms748_scan(
+        data, model, 0, 5, rtol=1e-8
+    )
+    observed_limit_default, expected_limits_default = results_default
+
+    # Force bounds_low to expand
+    observed_limit, expected_limits = pyhf.infer.intervals.upper_limits.toms748_scan(
+        data, model, 3, 5, rtol=1e-8
+    )
+
+    assert observed_limit == pytest.approx(observed_limit_default)
+    assert np.allclose(np.asarray(expected_limits), np.asarray(expected_limits_default))
+
+    # Force bounds_up to expand
+    observed_limit, expected_limits = pyhf.infer.intervals.upper_limits.toms748_scan(
+        data, model, 0, 1, rtol=1e-8
+    )
+    assert observed_limit == pytest.approx(observed_limit_default)
+    assert np.allclose(np.asarray(expected_limits), np.asarray(expected_limits_default))
+
+
+def test_upper_limit_against_auto(hypotest_args):
+    """
+    Test upper_limit linear scan and toms748_scan return similar results
+    """
+    _, data, model = hypotest_args
+    results_auto = pyhf.infer.intervals.upper_limits.toms748_scan(
+        data, model, 0, 5, rtol=1e-3
+    )
+    obs_auto, exp_auto = results_auto
+    results_linear = pyhf.infer.intervals.upper_limits.upper_limit(
+        data, model, scan=np.linspace(0, 5, 21)
+    )
+    obs_linear, exp_linear = results_linear
+    # Can't expect these to be much closer given the low granularity of the linear scan
+    assert obs_auto == pytest.approx(obs_linear, abs=0.1)
+    assert np.allclose(exp_auto, exp_linear, atol=0.1)
+
+
+def test_upper_limit(hypotest_args):
     """
     Check that the default return structure of pyhf.infer.hypotest is as expected
     """
     _, data, model = hypotest_args
-    results = pyhf.infer.intervals.upperlimit(data, model, scan=np.linspace(0, 5, 11))
+    scan = np.linspace(0, 5, 11)
+    results = pyhf.infer.intervals.upper_limits.upper_limit(data, model, scan=scan)
     assert len(results) == 2
     observed_limit, expected_limits = results
     assert observed_limit == pytest.approx(1.0262704738584554)
@@ -33,14 +115,24 @@ def test_upperlimit(tmpdir, hypotest_args):
         [0.65765653, 0.87999725, 1.12453992, 1.50243428, 2.09232927]
     )
 
+    # tighter relative tolerance needed for macos
+    results = pyhf.infer.intervals.upper_limits.upper_limit(data, model, rtol=1e-6)
+    assert len(results) == 2
+    observed_limit, expected_limits = results
+    assert observed_limit == pytest.approx(1.01156939)
+    assert expected_limits == pytest.approx(
+        [0.55988001, 0.75702336, 1.06234693, 1.50116923, 2.05078596]
+    )
+
 
-def test_upperlimit_with_kwargs(tmpdir, hypotest_args):
+def test_upper_limit_with_kwargs(hypotest_args):
     """
     Check that the default return structure of pyhf.infer.hypotest is as expected
     """
     _, data, model = hypotest_args
-    results = pyhf.infer.intervals.upperlimit(
-        data, model, scan=np.linspace(0, 5, 11), test_stat="qtilde"
+    scan = np.linspace(0, 5, 11)
+    results = pyhf.infer.intervals.upper_limits.upper_limit(
+        data, model, scan=scan, test_stat="qtilde"
     )
     assert len(results) == 2
     observed_limit, expected_limits = results
@@ -49,6 +141,30 @@ def test_upperlimit_with_kwargs(tmpdir, hypotest_args):
         [0.65765653, 0.87999725, 1.12453992, 1.50243428, 2.09232927]
     )
 
+    # linear_grid_scan
+    results = pyhf.infer.intervals.upper_limits.upper_limit(
+        data, model, scan=scan, return_results=True
+    )
+    assert len(results) == 3
+    observed_limit, expected_limits, (_scan, point_results) = results
+    assert observed_limit == pytest.approx(1.0262704738584554)
+    assert expected_limits == pytest.approx(
+        [0.65765653, 0.87999725, 1.12453992, 1.50243428, 2.09232927]
+    )
+    assert _scan.tolist() == scan.tolist()
+    assert len(_scan) == len(point_results)
+
+    # toms748_scan
+    results = pyhf.infer.intervals.upper_limits.upper_limit(
+        data, model, return_results=True, rtol=1e-6
+    )
+    assert len(results) == 3
+    observed_limit, expected_limits, (_scan, point_results) = results
+    assert observed_limit == pytest.approx(1.01156939)
+    assert expected_limits == pytest.approx(
+        [0.55988001, 0.75702336, 1.06234693, 1.50116923, 2.05078596]
+    )
+
 
 def test_mle_fit_default(tmpdir, hypotest_args):
     """
@@ -510,3 +626,19 @@ def test_teststat_nan_guard():
         test_poi, data, model, test_stat="qtilde", return_expected=True
     )
     assert all(~np.isnan(result) for result in test_results)
+
+
+# TODO: Remove after pyhf v0.9.0 is released
+def test_deprecated_upperlimit(hypotest_args):
+    with warnings.catch_warnings(record=True) as _warning:
+        # Cause all warnings to always be triggered
+        warnings.simplefilter("always")
+
+        _, data, model = hypotest_args
+        pyhf.infer.intervals.upperlimit(data, model, scan=np.linspace(0, 5, 11))
+        assert len(_warning) == 1
+        assert issubclass(_warning[-1].category, DeprecationWarning)
+        assert (
+            "pyhf.infer.intervals.upperlimit is deprecated in favor of pyhf.infer.intervals.upper_limits.upper_limit"
+            in str(_warning[-1].message)
+        )
diff --git a/tests/test_public_api_repr.py b/tests/test_public_api_repr.py
--- a/tests/test_public_api_repr.py
+++ b/tests/test_public_api_repr.py
@@ -125,7 +125,15 @@ def test_infer_calculators_public_api():
 
 
 def test_infer_intervals_public_api():
-    assert dir(pyhf.infer.intervals) == ["upperlimit"]
+    assert dir(pyhf.infer.intervals) == ["upper_limits.upper_limit"]
+
+
+def test_infer_intervals_upper_limit_public_api():
+    assert dir(pyhf.infer.intervals.upper_limits) == [
+        "linear_grid_scan",
+        "toms748_scan",
+        "upper_limit",
+    ]
 
 
 def test_infer_mle_public_api():