Repository: scikit-hep/pyhf

Issue:
Obtaining fit parameter labels
# Question

Is there a convenient way to obtain the labels corresponding to the parameters returned by `pyhf.infer.mle.fit()`? Setting up a fit is rather clean now with `infer`:
```
ws = pyhf.Workspace(spec)
model = ws.model()
data = ws.data(model)
pyhf.infer.mle.fit(data, model)
```
In comparison, the function in https://github.com/scikit-hep/pyhf/blob/master/docs/examples/notebooks/pullplot.ipynb to get the names of those parameters is difficult to parse for non-experts:
```
labels = [
        "{}[{}]".format(k, i) if model.config.param_set(k).n_parameters > 1 else k
        for k in model.config.par_order
        for i in range(model.config.param_set(k).n_parameters)
    ]
```
Is there a better way to get this list of parameter names, and if not, would it be possible to add this as a utility within `pyhf`?

# Relevant Issues and Pull Requests

not aware of any


Discussion:
> Is there a convenient way to obtain the labels corresponding to the parameters returned by `pyhf.infer.mle.fit()`?

@alexander-held as in using the public API? Not at the moment, but this seems reasonable enough that we can look into adding this. Though, as [`pyhf.infer.mle.fit(data, model)`](https://scikit-hep.org/pyhf/_generated/pyhf.infer.mle.fit.html) is just returning the best-fit model values it sounds like what you want is just a way to ask for what the model labels are, correct? If so, then this would probably go under the [`_ModelConfig`](https://scikit-hep.org/pyhf/_generated/pyhf.pdf._ModelConfig.html) API.
@alexander-held I guess in some sense it depends on what you mean by labels. `model.config.parameters` exists

```python
>>> import pyhf
>>> model = pyhf.simplemodels.hepdata_like(
...     signal_data=[12.0, 11.0], bkg_data=[50.0, 52.0], bkg_uncerts=[3.0, 7.0]
... )
>>> model.config.parameters
['mu', 'uncorr_bkguncrt']
```
though it doesn't make it clear here that ` 'uncorr_bkguncrt'` contains 2 parameters:

```python
>>> model.config.npars
3
>>> model.config.param_set("mu").n_parameters
1
>>> model.config.param_set("uncorr_bkguncrt").n_parameters
2
```
Just throwing out another way to do it. We can certainly provide a nicer API for it.

```python
    pars_pyhf = []
    for k, v in model.config.par_map.items():
        sl = v["slice"]
        npars = sl.stop - sl.start
        if npars > 1 or "staterror" in k:
            for i in range(npars):
                pars_pyhf.append("{}_{}".format(k, i))
        else:
            pars_pyhf.append(k)
```
It's a good question. In pyhf the only name we have are for vector-valued parameters sets. So far we didn't have names for the vector components themselves and this `{}_{}`.format(parset_name, index)` is more an ad-hoc name. 

If we wanted to introduce component-wise names as well we can think about it. E.g. could path-like `<parname>/<index>` or underscore. Also one would maybe need to distinguish between parameter sets that just happen to be size=1 (shapesys with 1 bin) but are potentially larger, versus paramsets that will always be size=1. In ROOT HiFa this is the distinctino between gammas and alphas


> `model.config.parameters`

In this context looking at the content of that might be more confusing than helpful to an inexperienced user, since the gammas for a channel show up as a single parameter. Let's say I run a fit with 1 POI and 1 channel with 2 bins, then pyhf.infer.mle.fit() gives me back the results of 3 parameters. `model.config.parameters` gives me back only two names though. I might then mistakenly think that the 2 gammas are correlated somehow internally. To figure out that one of them has this extra dimension with `n_parameters` requires a lot more understanding of the `pyhf` internals than anything else in this operation of getting a best fit result.

An API that provides the list would be awesome. This feature is useful to anyone who wants to study a fit in more detail.

Regarding the naming of parameters, I understand that internally there is no name for the different components of the vectors. Fixing this name within `pyhf` instead of letting the user come up with a convention themselves has the advantage of unifying the structure of these names for all users (at least those who use the API instead of coming up with their own convention). This is probably not crucial because for publication the parameters likely would be renamed anyway, but if all plots I look at showing fit results from workspaces from various people have the same naming pattern for e.g. gammas, it will be easier to parse visually what the gammas are.
Hi @alexander-held .

The reason I introduced this grouping initially was because I somewhat disliked that the grouping in ROOT HiFa is *only* apparent from the naming convention only, and nowhere made explicit (this also leads to some undocumented edge-cases where if you have a shapesys shared across channels with varying number of bins, which parameter corresponds to which bin becomes a bit arbitrary). It was also introduced a bit with an eye towards indeed cases where those parameters are a bit more closely connected (correlated) than in the uncorrelated staterror/shapesys cases.

 I agree though that it makes things a bit unusual / opaque

I was thinking whether e.g. a structured / jagged return value would make sense

I guess the `paramset` class could define some names

```
class paramset(object):
    def __init__(self, **kwargs):
        self.n_parameters = kwargs.pop('n_parameters')
        self.suggested_init = kwargs.pop('inits')
        self.suggested_bounds = kwargs.pop('bounds')
```


to 

```
class paramset(object):
    def __init__(self, **kwargs):
        self.n_parameters = kwargs.pop('n_parameters')
        self.suggested_init = kwargs.pop('inits')
        self.suggested_bounds = kwargs.pop('bounds')
        self.names = [str(i) for i in range(self.n_parameters)]
```

such that `'paramsetname'` and `'0' ` uniquely identify a scalar parameter of the likelihoood

from there one could get the full list of names via  


```
labels = [
"{}_{}".format(k, n) for k in model.config.par_order for n in model.config. param_set(k).names
]
 ```

but this would e.g. add a "_0" already for the "alphas" as well. e.g. "mu_0", "gamma_0", "gamma_1", "gamma_2". part of the reason why the expression you linked is a bit complex is in order to distinguish the naming between alphas (no suffix) and gammas (with suffix)


if we wanted to track this info (whether it's an "alpha" or "gamma" style paramset) we'd maybe need a `self.is_scalar` property or similar.
> In this context looking at the content of that might be more confusing than helpful to an inexperienced user, since the gammas for a channel show up as a single parameter.

Agreed. This was kinda my point with the follow up code snippet I had.

It seems like there is some good discussion here already, so we can probably start to think about how this could look in a `v0.5.1` release.
There's an initial implementation (#1536) for now as `model.config.par_names()` that can be expanded or refactored as needed. We likely won't settle on that API for a final form anyway. Note that the default propagation to minuit is the original format mentioned at the start of the issue `{name}[{index}]` (but of course, `index` is somewhat vague here - but is meant to implicitly refer to the n parameters in the slices. For now, the assumption is that the slices are contiguous but this might not always be the case in the future.
Do you expect the label format to change again? Since the propagation to MINUIT has the default format, there are two ways of handling this in my mind when using `pyhf`:
- completely ignore the new `model.config.par_names()` and handle the translation to the default names (previously `x0` etc, now the default `model.config.par_names()`) needed by MINOS internally (as done currently in `cabinetry`),
- adopt the new format consistently everywhere without using the `fstring` argument.

The use of `fstring` implies that users would need to either be aware that the names they are seeing (e.g. in plots) do not correspond to what they need to pass to MINOS, or some conversion magic would need to be done.

Wrapping the functionality of `model.config.par_names()` via external libraries to provide a consistent API (if you expect it to change again) could work, but has the big disadvantage of having multiple implementations to get parameter names that may disagree, and then it is unclear to users which format to use.

I think it would be very useful to be able to rely on some format (e.g. the newly implemented one) as being stable.
What do you think about adding the `[0]` suffix also for modifiers that happen to be scalar, but only because the channel has a single bin? In models with a mix of single- and multi-bin channels, it perhaps looks a bit surprising to have a mix of modifiers of the same type with and without that suffix:
```txt
staterror_Signal_region
staterror_Control_region[0]
staterror_Control_region[1]
staterror_Control_region[2]
```
> The use of `fstring` implies that users would need to either be aware that the names they are seeing (e.g. in plots) do not correspond to what they need to pass to MINOS, or some conversion magic would need to be done.
> 
> Wrapping the functionality of `model.config.par_names()` via external libraries to provide a consistent API (if you expect it to change again) could work, but has the big disadvantage of having multiple implementations to get parameter names that may disagree, and then it is unclear to users which format to use.

It's a fair point. One other thought is that the `fstring` was some member variable instead that could be configured instead of relying on any sort of default implementation. But I've been thinking about dropping it entirely and being a bit opinionated here. `pyhf` tends to be really flexible on a lot of things and allow the user to do a lot, but maybe this isn't needed for this area.

> What do you think about adding the `[0]` suffix also for modifiers that happen to be scalar, but only because the channel has a single bin? In models with a mix of single- and multi-bin channels, it perhaps looks a bit surprising to have a mix of modifiers of the same type with and without that suffix:

Ahh, this needs to be handled still. I have a way of fixing this, but need to propagate some new information to the modifiers to indicate if they're scalar or vector.
> One other thought is that the fstring was some member variable instead that could be configured instead of relying on any sort of default implementation.

Making it a member variable feels a bit clunky, and seems different in design from the rest of `pyhf` to me.

> But I've been thinking about dropping it entirely and being a bit opinionated here.

I like the idea of dropping it completely. A small function can still be written by users to achieve this, but within `pyhf` all naming would be consistent and unambiguous. If users want things like replacing text with LaTeX and similar operations that may be needed for publication-quality plots, the flexibility achieved via a manual implementation may be needed anyway.

Resolving patch:
diff --git a/src/pyhf/pdf.py b/src/pyhf/pdf.py
--- a/src/pyhf/pdf.py
+++ b/src/pyhf/pdf.py
@@ -319,13 +319,10 @@ def par_slice(self, name):
         """
         return self.par_map[name]['slice']
 
-    def par_names(self, fstring='{name}[{index}]'):
+    def par_names(self):
         """
         The names of the parameters in the model including binned-parameter indexing.
 
-        Args:
-            fstring (:obj:`str`): Format string for the parameter names using ``name`` and ``index`` variables. Default: ``'{name}[{index}]'``.
-
         Returns:
             :obj:`list`: Names of the model parameters.
 
@@ -336,18 +333,16 @@ def par_names(self, fstring='{name}[{index}]'):
             ... )
             >>> model.config.par_names()
             ['mu', 'uncorr_bkguncrt[0]', 'uncorr_bkguncrt[1]']
-            >>> model.config.par_names(fstring='{name}_{index}')
-            ['mu', 'uncorr_bkguncrt_0', 'uncorr_bkguncrt_1']
         """
         _names = []
         for name in self.par_order:
-            _npars = self.param_set(name).n_parameters
-            if _npars == 1:
+            param_set = self.param_set(name)
+            if param_set.is_scalar:
                 _names.append(name)
                 continue
 
             _names.extend(
-                [fstring.format(name=name, index=idx) for idx in range(_npars)]
+                [f'{name}[{index}]' for index in range(param_set.n_parameters)]
             )
         return _names
 


Test patch:
diff --git a/tests/test_optim.py b/tests/test_optim.py
--- a/tests/test_optim.py
+++ b/tests/test_optim.py
@@ -581,7 +581,7 @@ def test_minuit_param_names(mocker):
     data = [10] + pdf.config.auxdata
     _, result = pyhf.infer.mle.fit(data, pdf, return_result_obj=True)
     assert 'minuit' in result
-    assert result.minuit.parameters == ('mu', 'uncorr_bkguncrt')
+    assert result.minuit.parameters == ('mu', 'uncorr_bkguncrt[0]')
 
     pdf.config.par_names = mocker.Mock(return_value=None)
     _, result = pyhf.infer.mle.fit(data, pdf, return_result_obj=True)
diff --git a/tests/test_pdf.py b/tests/test_pdf.py
--- a/tests/test_pdf.py
+++ b/tests/test_pdf.py
@@ -890,3 +890,34 @@ def test_reproducible_model_spec():
         {'bounds': [[0, 5]], 'inits': [1], 'name': 'mu'}
     ]
     assert pyhf.Model(model_from_ws.spec)
+
+
+def test_par_names_scalar_nonscalar():
+    """
+    Testing to ensure that nonscalar parameters are still indexed, even if
+    n_parameters==1.
+    """
+    spec = {
+        'channels': [
+            {
+                'name': 'channel',
+                'samples': [
+                    {
+                        'name': 'goodsample',
+                        'data': [1.0],
+                        'modifiers': [
+                            {'type': 'normfactor', 'name': 'scalar', 'data': None},
+                            {'type': 'shapesys', 'name': 'nonscalar', 'data': [1.0]},
+                        ],
+                    },
+                ],
+            }
+        ]
+    }
+
+    model = pyhf.Model(spec, poi_name="scalar")
+    assert model.config.par_order == ["scalar", "nonscalar"]
+    assert model.config.par_names() == [
+        'scalar',
+        'nonscalar[0]',
+    ]